Bug 1414224 - Introspection Fails on all nodes using dracclient
Summary: Introspection Fails on all nodes using dracclient
Keywords:
Status: CLOSED WONTFIX
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openwsman
Version: 9.0 (Mitaka)
Hardware: Unspecified
OS: Unspecified
high
unspecified
Target Milestone: ---
: ---
Assignee: RHOS Maint
QA Contact: nlevinki
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-01-18 05:40 UTC by Md Nadeem
Modified: 2020-12-14 08:01 UTC (History)
17 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2018-02-14 14:55:17 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Bugzilla 1463224 0 medium CLOSED iDRAC get/set power state fails (OSP-9) 2021-02-22 00:41:40 UTC
Red Hat Knowledge Base (Solution) 3023991 0 None None None 2017-05-05 09:30:37 UTC

Description Md Nadeem 2017-01-18 05:40:28 UTC
Created attachment 1242028 [details]
Logs

Description of problem:

Inspection fails with RHOSP9 for all node with the error message "Introspection didn't finish for nodes ... "

However, it gets pass with RHOSP10

I have attached, all required logs and screenshot. Please find it.

Environment:

- Undercloud(VM on ESXi6.5) : RHEL7.2
- Bare metal node using Overcloud: DELL R230 (BIOS v1.4.5, Firmware v2.41.40.40, Lifecycle Controller Firmware v2.41.40.40) * 6EA

Note: As per below reference, DELL R230 server is certified with both OSP9 & OSP10
>> https://access.redhat.com/ecosystem/hardware/2064063#note3

Detailed Analysis:

After analyzing the logs, i gets below findings,

Overcloud node gets ip: X.X.X.10/24, and a set of mac addresses[u'X:X:X:a4:a7:d7', u'X:X:X:a4:a5:eb', u'X:X:X:a4:a8:af', u'X:X:X:a4:a8:e7', u'X:X:X:a4:a4:97'] available for the overcloud node.

Overcloud node gets bmc_address': [u'X.X.X.11'], u'mac' for bmc: [u'X:X:X:a4:a7:bf']

***Inspection of overcloud started at 

>> Jan 11 20:49:23 director.naimnetworks.com ironic-inspector[9080]: 2017-01-11 20:49:23.891 9080 INFO ironic_inspector.introspect [-] [node: 684bb2e8-3fb6-4e30-9380-1493bd360afd] Introspection started successfully

***However, just after starting of introspection, ironic-conductor logged a strong warning for adding the certificate for HTTPS connection as shown below

>> Jan 11 20:49:24 director.naimnetworks.com ironic-conductor[9401]: /usr/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py:821: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.org/en/latest/security.html

***After analyzing, tcp dump i find all the communication happens between undercloud and overcloud node over HTTP except the request that goes to [https://X.X.X.11:443/wsman] as shown below.

>> Jan 11 20:49:24 director.naimnetworks.com ironic-conductor[9401]: 2017-01-11 20:49:24.836 9401 DEBUG dracclient.wsman [req-18a19a38-4a45-4842-b047-f996c6412bbd - - - - -] Sending request to https://X.X.X.11:443/wsman: <s:Envelope xmlns:wsa="http://schemas.xmlsoap.org/ws/2004/08/addressing" xmlns:s="http://www.w3.org/2003/05/soap-envelope" xmlns:wsman="http://schemas.dmtf.org/wbem/wsman/1/wsman.xsd"><s:Header><wsa:To s:mustUnderstand="true">https://X.X.X.11:443/wsman</wsa:To><wsman:ResourceURI s:mustUnderstand="true">http://schemas.dell.com/wbem/wscim/1/cim-schema/2/DCIM_LifecycleJob</wsman:ResourceURI><wsa:MessageID s:mustUnderstand="true">uuid:3d576955-1de0-4c6d-92f7-d8d88413d14e</wsa:MessageID><wsa:ReplyTo><wsa:Address>http://schemas.xmlsoap.org/ws/2004/08/addressing/role/anonymous</wsa:Address></wsa:ReplyTo><wsa:Action s:mustUnderstand="true">http://schemas.xmlsoap.org/ws/2004/09/enumeration/Enumerate</wsa:Action></s:Header><s:Body><wsen:Enumerate xmlns:wsen="http://schemas.xmlsoap.org/ws/2004/09/enumeration"><wsman:Filter Dialect="http://schemas.dmtf.org/wbem/cql/1/dsp0202.pdf">select * from DCIM_LifecycleJob where Name != "CLEARALL" and JobStatus != "Reboot Completed" and JobStatus != "Completed" and JobStatus != "Completed with Errors" and JobStatus != "Failed"</wsman:Filter><wsman:OptimizeEnumeration/><wsman:MaxElements>100</wsman:MaxElements></wsen:Enumerate></s:Body></s:Envelope> _do_request /usr/lib/python2.7/site-packages/dracclient/wsman.py:60 



***Ironic driver module and draclient modules throw below errors

>>2017-01-11 15:11:16.074 2253 ERROR dracclient.wsman SSLError: EOF occurred in violation of protocol (_ssl.c:579)
>> 2017-01-11 15:11:16.083 2253 ERROR ironic.drivers.modules.drac.power [req-014548b4-3451-40a1-b65f-bc0dd0f77db6 admin - - - -] DRAC driver failed to get power state for node 79ad4497-b759-4d06-9340-c5bf59a45145. Reason: WSMan request failed.
2017-01-11 15:11:16.084 2253 WARNING ironic.conductor.manager [req-014548b4-3451-40a1-b65f-bc0dd0f77db6 admin - - - -] During sync_power_state, could not get power state for node 79ad4497-b759-4d06-9340-c5bf59a45145, attempt 1 of 3. Error: DRAC operation failed. Reason: WSMan request failed.
>> dracclient.wsman SSLError: EOF occurred in violation of protocol (_ssl.c:579)


*** Ironic-inspector logged authentication error as mentioned below

>> Jan 11 14:53:29 director.naimnetworks.com ironic-inspector[28360]: 2017-01-11 14:53:29.813 28360 ERROR ironic_inspector.main AmbiguousAuthSystem: Must provide Keystone credentials or user-defined endpoint and token, error was: The request you have made requires authentication. (HTTP 401) (Request-ID: req-de692231-3474-4595-af47-ae05030d24f3)

Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:

Introspection didn't finish for nodes

Expected results:

Introspection should successfully completed for all nodes.

Additional info:

The same issue occurs while try to introspect a single server also.

Comment 1 Sergii Mykhailushko 2017-04-11 13:17:10 UTC
Customer is experiencing similar problem when running introspection with mistral workflow tool in OSP10. Probably he's hitting the same bug in his environment:

$ mistral execution-list | grep "ERROR";
| 2a7c2499-03e7-4c04-a1a4-4a23f5d4bb52 | 70db247b-1c8d-414a-ae98-595545183399 | tripleo.baremetal.v1.introspect                         | sub-workflow execution | b5f64f1d-f79b-4a0a-b6cb-dbdf57c1e55a | ERROR   | Failure caused by error i... | 2017-03-21 11:08:08 | 2017-03-21 11:19:32 |
| 5da548ed-ba25-4e6b-96e6-8dc0ca7dcacc | 70db247b-1c8d-414a-ae98-595545183399 | tripleo.baremetal.v1.introspect                         | sub-workflow execution | 4b513c7c-0566-4ae8-879a-2e2ff2e51f1f | ERROR   | Failure caused by error i... | 2017-03-21 13:59:02 | 2017-03-21 14:08:56 |
| cc2b7e9e-7286-425d-bedd-aa77ddc54185 | 70db247b-1c8d-414a-ae98-595545183399 | tripleo.baremetal.v1.introspect                         | sub-workflow execution | b8204bfa-6d27-4700-93c4-d05296a61654 | ERROR   | Failure caused by error i... | 2017-03-21 15:17:54 | 2017-03-21 15:27:25 |
| 9962345f-97d2-47f5-ab1b-0c857e8ba305 | 70db247b-1c8d-414a-ae98-595545183399 | tripleo.baremetal.v1.introspect                         | sub-workflow execution | 82726201-9cf1-4ce2-9f6e-2fd39104d2f8 | ERROR   | Failure caused by error i... | 2017-03-21 16:10:15 | 2017-03-21 16:19:51 |


$ mistral execution-get 9962345f-97d2-47f5-ab1b-0c857e8ba305
+-------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field             | Value                                                                                                                                                                                                 |
+-------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| ID                | 9962345f-97d2-47f5-ab1b-0c857e8ba305                                                                                                                                                                  |
| Workflow ID       | 70db247b-1c8d-414a-ae98-595545183399                                                                                                                                                                  |
| Workflow name     | tripleo.baremetal.v1.introspect                                                                                                                                                                       |
| Description       | sub-workflow execution                                                                                                                                                                                |
| Task Execution ID | 82726201-9cf1-4ce2-9f6e-2fd39104d2f8                                                                                                                                                                  |
| State             | ERROR                                                                                                                                                                                                 |
| State info        | Failure caused by error in tasks: wait_for_introspection_to_finish                                                                                                                                    |
|                   |                                                                                                                                                                                                       |
|                   |   wait_for_introspection_to_finish [task_ex_id=6415c7f2-451a-483e-8b4c-74e4bcd2b277] -> Failed to run action [action_ex_id=2ffee968-eae8-499c-aeb9-70e4c81fb1ab, action_cls='<class                   |
|                   | 'mistral.actions.action_factory.BaremetalIntrospectionAction'>', attributes='{u'client_method_name': u'wait_for_finish'}', params='{u'max_retries': 3600, u'retry_interval': 10, u'uuids':            |
|                   | [u'ac893774-9a1a-4d7c-be8e-8971cd9a5c28', u'20789f81-13f2-42ca-b8b6-c1655c6a2d76', u'4128fc00-8340-48a0-a4c9-d3d3ac1f3ca5', u'd9279bc8-17bb-4227-8e5c-6305f90c4d4e', u'04366019-a2f4-46ca-            |
|                   | 9f09-77df630f18f8', u'1b522322-9f79-4e5a-b1d3-c2ab329f8bc0', u'ef0ea07e-f1ff-4942-9d23-f5798f5cdf19', u'a0eea23b-aaaa-4386-96fe-f13fef6a04fb']}']                                                     |
|                   |  BaremetalIntrospectionAction.wait_for_finish failed: <class 'keystoneauth1.exceptions.connection.ConnectFailure'>: Connection failure that may be retried.                                           |
|                   |     [action_ex_id=2ffee968-eae8-499c-aeb9-70e4c81fb1ab, idx=0]: Failed to run action [action_ex_id=2ffee968-eae8-499c-aeb9-70e4c81fb1ab, action_cls='<class                                           |
|                   | 'mistral.actions.action_factory.BaremetalIntrospectionAction'>', attributes='{u'client_method_name': u'wait_for_finish'}', params='{u'max_retries': 3600, u'retry_interval': 10, u'uuids':            |
|                   | [u'ac893774-9a1a-4d7c-be8e-8971cd9a5c28', u'20789f81-13f2-42ca-b8b6-c1655c6a2d76', u'4128fc00-8340-48a0-a4c9-d3d3ac1f3ca5', u'd9279bc8-17bb-4227-8e5c-6305f90c4d4e', u'04366019-a2f4-46ca-            |
|                   | 9f09-77df630f18f8', u'1b522322-9f79-4e5a-b1d3-c2ab329f8bc0', u'ef0ea07e-f1ff-4942-9d23-f5798f5cdf19', u'a0eea23b-aaaa-4386-96fe-f13fef6a04fb']}']                                                     |
|                   |  BaremetalIntrospectionAction.wait_for_finish failed: <class 'keystoneauth1.exceptions.connection.ConnectFailure'>: Connection failure that may be retried.                                           |
|                   |                                                                                                                                                                                                       |
| Created at        | 2017-03-21 16:10:15                                                                                                                                                                                   |
| Updated at        | 2017-03-21 16:19:51                                                                                                                                                                                   |
+-------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

$ mistral execution-get-output 9962345f-97d2-47f5-ab1b-0c857e8ba305
{
    "result": "Failure caused by error in tasks: wait_for_introspection_to_finish\n\n  wait_for_introspection_to_finish [task_ex_id=6415c7f2-451a-483e-8b4c-74e4bcd2b277] -> Failed to run action [action_ex_id=2ffee968-eae8-499c-aeb9-70e4c81fb1ab, action_cls='<class 'mistral.actions.action_factory.BaremetalIntrospectionAction'>', attributes='{u'client_method_name': u'wait_for_finish'}', params='{u'max_retries': 3600, u'retry_interval': 10, u'uuids': [u'ac893774-9a1a-4d7c-be8e-8971cd9a5c28', u'20789f81-13f2-42ca-b8b6-c1655c6a2d76', u'4128fc00-8340-48a0-a4c9-d3d3ac1f3ca5', u'd9279bc8-17bb-4227-8e5c-6305f90c4d4e', u'04366019-a2f4-46ca-9f09-77df630f18f8', u'1b522322-9f79-4e5a-b1d3-c2ab329f8bc0', u'ef0ea07e-f1ff-4942-9d23-f5798f5cdf19', u'a0eea23b-aaaa-4386-96fe-f13fef6a04fb']}']\n BaremetalIntrospectionAction.wait_for_finish failed: <class 'keystoneauth1.exceptions.connection.ConnectFailure'>: Connection failure that may be retried.\n    [action_ex_id=2ffee968-eae8-499c-aeb9-70e4c81fb1ab, idx=0]: Failed to run action [action_ex_id=2ffee968-eae8-499c-aeb9-70e4c81fb1ab, action_cls='<class 'mistral.actions.action_factory.BaremetalIntrospectionAction'>', attributes='{u'client_method_name': u'wait_for_finish'}', params='{u'max_retries': 3600, u'retry_interval': 10, u'uuids': [u'ac893774-9a1a-4d7c-be8e-8971cd9a5c28', u'20789f81-13f2-42ca-b8b6-c1655c6a2d76', u'4128fc00-8340-48a0-a4c9-d3d3ac1f3ca5', u'd9279bc8-17bb-4227-8e5c-6305f90c4d4e', u'04366019-a2f4-46ca-9f09-77df630f18f8', u'1b522322-9f79-4e5a-b1d3-c2ab329f8bc0', u'ef0ea07e-f1ff-4942-9d23-f5798f5cdf19', u'a0eea23b-aaaa-4386-96fe-f13fef6a04fb']}']\n BaremetalIntrospectionAction.wait_for_finish failed: <class 'keystoneauth1.exceptions.connection.ConnectFailure'>: Connection failure that may be retried.\n"
}

$ mistral execution-get cc2b7e9e-7286-425d-bedd-aa77ddc54185
+-------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field             | Value                                                                                                                                                                                                 |
+-------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| ID                | cc2b7e9e-7286-425d-bedd-aa77ddc54185                                                                                                                                                                  |
| Workflow ID       | 70db247b-1c8d-414a-ae98-595545183399                                                                                                                                                                  |
| Workflow name     | tripleo.baremetal.v1.introspect                                                                                                                                                                       |
| Description       | sub-workflow execution                                                                                                                                                                                |
| Task Execution ID | b8204bfa-6d27-4700-93c4-d05296a61654                                                                                                                                                                  |
| State             | ERROR                                                                                                                                                                                                 |
| State info        | Failure caused by error in tasks: wait_for_introspection_to_finish                                                                                                                                    |
|                   |                                                                                                                                                                                                       |
|                   |   wait_for_introspection_to_finish [task_ex_id=a43444e1-ff02-4c61-bfec-12daabdc5517] -> Failed to run action [action_ex_id=d3449142-70d7-4c2e-86bd-1b2407158a12, action_cls='<class                   |
|                   | 'mistral.actions.action_factory.BaremetalIntrospectionAction'>', attributes='{u'client_method_name': u'wait_for_finish'}', params='{u'max_retries': 3600, u'retry_interval': 10, u'uuids':            |
|                   | [u'ac893774-9a1a-4d7c-be8e-8971cd9a5c28', u'f4b1f1d8-ba40-42aa-8430-b8249265535e', u'20789f81-13f2-42ca-b8b6-c1655c6a2d76', u'4128fc00-8340-48a0-a4c9-d3d3ac1f3ca5', u'd9279bc8-17bb-4227-8e5c-       |
|                   | 6305f90c4d4e', u'04366019-a2f4-46ca-9f09-77df630f18f8', u'1b522322-9f79-4e5a-b1d3-c2ab329f8bc0', u'ef0ea07e-f1ff-4942-9d23-f5798f5cdf19', u'a0eea23b-aaaa-4386-96fe-f13fef6a04fb']}']                 |
|                   |  BaremetalIntrospectionAction.wait_for_finish failed: <class 'keystoneauth1.exceptions.connection.ConnectFailure'>: Connection failure that may be retried.                                           |
|                   |     [action_ex_id=d3449142-70d7-4c2e-86bd-1b2407158a12, idx=0]: Failed to run action [action_ex_id=d3449142-70d7-4c2e-86bd-1b2407158a12, action_cls='<class                                           |
|                   | 'mistral.actions.action_factory.BaremetalIntrospectionAction'>', attributes='{u'client_method_name': u'wait_for_finish'}', params='{u'max_retries': 3600, u'retry_interval': 10, u'uuids':            |
|                   | [u'ac893774-9a1a-4d7c-be8e-8971cd9a5c28', u'f4b1f1d8-ba40-42aa-8430-b8249265535e', u'20789f81-13f2-42ca-b8b6-c1655c6a2d76', u'4128fc00-8340-48a0-a4c9-d3d3ac1f3ca5', u'd9279bc8-17bb-4227-8e5c-       |
|                   | 6305f90c4d4e', u'04366019-a2f4-46ca-9f09-77df630f18f8', u'1b522322-9f79-4e5a-b1d3-c2ab329f8bc0', u'ef0ea07e-f1ff-4942-9d23-f5798f5cdf19', u'a0eea23b-aaaa-4386-96fe-f13fef6a04fb']}']                 |
|                   |  BaremetalIntrospectionAction.wait_for_finish failed: <class 'keystoneauth1.exceptions.connection.ConnectFailure'>: Connection failure that may be retried.                                           |
|                   |                                                                                                                                                                                                       |
| Created at        | 2017-03-21 15:17:54                                                                                                                                                                                   |
| Updated at        | 2017-03-21 15:27:25                                                                                                                                                                                   |
+-------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

$ mistral execution-get-output cc2b7e9e-7286-425d-bedd-aa77ddc54185
{
    "result": "Failure caused by error in tasks: wait_for_introspection_to_finish\n\n  wait_for_introspection_to_finish [task_ex_id=a43444e1-ff02-4c61-bfec-12daabdc5517] -> Failed to run action [action_ex_id=d3449142-70d7-4c2e-86bd-1b2407158a12, action_cls='<class 'mistral.actions.action_factory.BaremetalIntrospectionAction'>', attributes='{u'client_method_name': u'wait_for_finish'}', params='{u'max_retries': 3600, u'retry_interval': 10, u'uuids': [u'ac893774-9a1a-4d7c-be8e-8971cd9a5c28', u'f4b1f1d8-ba40-42aa-8430-b8249265535e', u'20789f81-13f2-42ca-b8b6-c1655c6a2d76', u'4128fc00-8340-48a0-a4c9-d3d3ac1f3ca5', u'd9279bc8-17bb-4227-8e5c-6305f90c4d4e', u'04366019-a2f4-46ca-9f09-77df630f18f8', u'1b522322-9f79-4e5a-b1d3-c2ab329f8bc0', u'ef0ea07e-f1ff-4942-9d23-f5798f5cdf19', u'a0eea23b-aaaa-4386-96fe-f13fef6a04fb']}']\n BaremetalIntrospectionAction.wait_for_finish failed: <class 'keystoneauth1.exceptions.connection.ConnectFailure'>: Connection failure that may be retried.\n    [action_ex_id=d3449142-70d7-4c2e-86bd-1b2407158a12, idx=0]: Failed to run action [action_ex_id=d3449142-70d7-4c2e-86bd-1b2407158a12, action_cls='<class 'mistral.actions.action_factory.BaremetalIntrospectionAction'>', attributes='{u'client_method_name': u'wait_for_finish'}', params='{u'max_retries': 3600, u'retry_interval': 10, u'uuids': [u'ac893774-9a1a-4d7c-be8e-8971cd9a5c28', u'f4b1f1d8-ba40-42aa-8430-b8249265535e', u'20789f81-13f2-42ca-b8b6-c1655c6a2d76', u'4128fc00-8340-48a0-a4c9-d3d3ac1f3ca5', u'd9279bc8-17bb-4227-8e5c-6305f90c4d4e', u'04366019-a2f4-46ca-9f09-77df630f18f8', u'1b522322-9f79-4e5a-b1d3-c2ab329f8bc0', u'ef0ea07e-f1ff-4942-9d23-f5798f5cdf19', u'a0eea23b-aaaa-4386-96fe-f13fef6a04fb']}']\n BaremetalIntrospectionAction.wait_for_finish failed: <class 'keystoneauth1.exceptions.connection.ConnectFailure'>: Connection failure that may be retried.\n"
}

The following can be observed in the log:

---
17750:Mar 21 16:17:19 ospd ironic-inspector: 2017-03-21 16:17:19.576 923 DEBUG ironic_inspector.process [-] [node: MAC a0:36:9f:d3:f4:b0 BMC 192.0.2.129] Running pre-processing hook ramdisk_error _run_pre_hooks 
/usr/lib/python2.7/site-packages/ironic_inspector/process.py:119
17790:Mar 21 16:17:20 ospd ironic-inspector: 2017-03-21 16:17:20.006 923 DEBUG ironic_inspector.process [-] [node: ac893774-9a1a-4d7c-be8e-8971cd9a5c28 MAC a0:36:9f:d3:f4:b0 BMC 192.0.2.129] Running post-process
ing hook ramdisk_error _run_post_hooks /usr/lib/python2.7/site-packages/ironic_inspector/process.py:262
17861:Mar 21 16:17:22 ospd ironic-inspector: [{u'rotational': False, u'vendor': u'ATA', u'name': u'/dev/sda', u'wwn_vendor_extension': None, u'wwn_with_extension': u'0x55cd2e414d4d43d0', u'model': u'INTEL SSDSC2
BB12', u'wwn': u'0x55cd2e414d4d43d0', u'serial': u'PHWA640601HW120CGN', u'size': 120034123776}], u'boot': {u'current_boot_mode': u'bios', u'pxe_interface': u'a0:36:9f:d3:f1:f4'}, u'system_vendor': {u'serial_numb
er': u'7SRCZG2', u'product_name': u'PowerEdge C6320', u'manufacturer': u'Dell Inc.'}, u'memory': {u'physical_mb': 262144, u'total': 270375514112}, u'cpu': {u'count': 64, u'frequency': u'1202.414', u'flags': [u'f
pu', u'vme', u'de', u'pse', u'tsc', u'msr', u'pae', u'mce', u'cx8', u'apic', u'sep', u'mtrr', u'pge', u'mca', u'cmov', u'pat', u'pse36', u'clflush', u'dts', u'acpi', u'mmx', u'fxsr', u'sse', u'sse2', u'ss', u'ht
', u'tm', u'pbe', u'syscall', u'nx', u'pdpe1gb', u'rdtscp', u'lm', u'constant_tsc', u'arch_perfmon', u'pebs', u'bts', u'rep_good', u'nopl', u'xtopology', u'nonstop_tsc', u'aperfmperf', u'eagerfpu', u'pni', u'pcl
mulqdq', u'dtes64', u'monitor', u'ds_cpl', u'vmx', u'smx', u'est', u'tm2', u'ssse3', u'fma', u'cx16', u'xtpr', u'pdcm', u'pcid', u'dca', u'sse4_1', u'sse4_2', u'x2apic', u'movbe', u'popcnt', u'tsc_deadline_timer
', u'aes', u'xsave', u'avx', u'f16c', u'rdrand', u'lahf_lm', u'abm', u'3dnowprefetch', u'ida', u'arat', u'epb', u'pln', u'pts', u'dtherm', u'intel_pt', u'tpr_shadow', u'vnmi', u'flexpriority', u'ept', u'vpid', u
'fsgsbase', u'tsc_adjust', u'bmi1', u'hle', u'avx2', u'smep', u'bmi2', u'erms', u'invpcid', u'rtm', u'cqm', u'rdseed', u'adx', u'smap', u'xsaveopt', u'cqm_llc', u'cqm_occup_llc', u'cqm_mbm_total', u'cqm_mbm_loca
l'], u'model_name': u'Intel(R) Xeon(R) CPU E5-2683 v4 @ 2.10GHz', u'architecture': u'x86_64'}}, u'error': None, u'local_gb': 110, u'logs': '<hidden>'} api_continue /usr/lib/python2.7/site-packages/ironic_inspect
or/main.py:180
17862:Mar 21 16:17:22 ospd ironic-inspector: 2017-03-21 16:17:22.090 923 DEBUG ironic_inspector.process [-] [node: MAC a0:36:9f:d3:f1:f4 BMC 192.0.2.132] Running pre-processing hook ramdisk_error _run_pre_hooks 
/usr/lib/python2.7/site-packages/ironic_inspector/process.py:119
17891:Mar 21 16:17:22 ospd ironic-inspector: 2017-03-21 16:17:22.217 923 DEBUG ironic_inspector.process [-] [node: 4128fc00-8340-48a0-a4c9-d3d3ac1f3ca5 MAC a0:36:9f:d3:f1:f4 BMC 192.0.2.132] Running post-process
ing hook ramdisk_error _run_post_hooks /usr/lib/python2.7/site-packages/ironic_inspector/process.py:262
18031:Mar 21 16:17:27 ospd ironic-conductor: 2017-03-21 16:17:27.199 9729 ERROR dracclient.wsman [req-a9c58c37-1e59-4395-8879-45942b004894 - - - - -] Request failed
18032:Mar 21 16:17:27 ospd ironic-conductor: 2017-03-21 16:17:27.199 9729 ERROR dracclient.wsman Traceback (most recent call last):
18033:Mar 21 16:17:27 ospd ironic-conductor: 2017-03-21 16:17:27.199 9729 ERROR dracclient.wsman   File "/usr/lib/python2.7/site-packages/dracclient/wsman.py", line 67, in _do_request
18034:Mar 21 16:17:27 ospd ironic-conductor: 2017-03-21 16:17:27.199 9729 ERROR dracclient.wsman     verify=False)
18035:Mar 21 16:17:27 ospd ironic-conductor: 2017-03-21 16:17:27.199 9729 ERROR dracclient.wsman   File "/usr/lib/python2.7/site-packages/requests/api.py", line 111, in post
18036:Mar 21 16:17:27 ospd ironic-conductor: 2017-03-21 16:17:27.199 9729 ERROR dracclient.wsman     return request('post', url, data=data, json=json, **kwargs)
18037:Mar 21 16:17:27 ospd ironic-conductor: 2017-03-21 16:17:27.199 9729 ERROR dracclient.wsman   File "/usr/lib/python2.7/site-packages/requests/api.py", line 57, in request
18038:Mar 21 16:17:27 ospd ironic-conductor: 2017-03-21 16:17:27.199 9729 ERROR dracclient.wsman     return session.request(method=method, url=url, **kwargs)
18039:Mar 21 16:17:27 ospd ironic-conductor: 2017-03-21 16:17:27.199 9729 ERROR dracclient.wsman   File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 475, in request
18040:Mar 21 16:17:27 ospd ironic-conductor: 2017-03-21 16:17:27.199 9729 ERROR dracclient.wsman     resp = self.send(prep, **send_kwargs)
18041:Mar 21 16:17:27 ospd ironic-conductor: 2017-03-21 16:17:27.199 9729 ERROR dracclient.wsman   File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 585, in send
18042:Mar 21 16:17:27 ospd ironic-conductor: 2017-03-21 16:17:27.199 9729 ERROR dracclient.wsman     r = adapter.send(request, **kwargs)
18043:Mar 21 16:17:27 ospd ironic-conductor: 2017-03-21 16:17:27.199 9729 ERROR dracclient.wsman   File "/usr/lib/python2.7/site-packages/requests/adapters.py", line 453, in send
18044:Mar 21 16:17:27 ospd ironic-conductor: 2017-03-21 16:17:27.199 9729 ERROR dracclient.wsman     raise ConnectionError(err, request=request)
18045:Mar 21 16:17:27 ospd ironic-conductor: 2017-03-21 16:17:27.199 9729 ERROR dracclient.wsman ConnectionError: ('Connection aborted.', BadStatusLine("''",))
18046:Mar 21 16:17:27 ospd ironic-conductor: 2017-03-21 16:17:27.199 9729 ERROR dracclient.wsman
18047:Mar 21 16:17:27 ospd ironic-conductor: 2017-03-21 16:17:27.202 9729 ERROR ironic.drivers.modules.drac.power [req-a9c58c37-1e59-4395-8879-45942b004894 - - - - -] DRAC driver failed to get power state for no
de ea88388f-8c83-498e-9299-e23dfe7bcd56. Reason: WSMan request failed.
---

Comment 4 Dmitry Tantsur 2017-05-09 07:44:04 UTC
Hi!

In the later releases we've moved away from openwsman, this is why it works for you with OSP 10. I recommend trying the pxe_ipmitool driver instead in your release. Note that https://github.com/OpenNMS/wsman is a Java library, it's unlikely to be related.

Sergii, your problem looks different and at first glance is related to either Keystone or ironic-inspector services being down. Please check their logs.

Comment 7 Bob Fournier 2017-09-22 13:50:43 UTC
There are known issues using dracclient in OSP-9, see upstream bug related to get/set power - https://bugs.launchpad.net/python-dracclient/+bug/1667088, and the BZ referencing  it https://bugzilla.redhat.com/show_bug.cgi?id=1463224.  Its currently recommended to use pxe_ipmitool in OSP-9.

Comment 8 Dmitry Tantsur 2017-10-25 12:26:02 UTC
Can someone having these problems please try:

1. updating the firmware of their Drac's,
2. installing python-dracclient from a more recent release (OSP 10, for example),
3. try pxe_ipmitool driver instead of pxe_ilo?

OSP 9 has python-dracclient 0.0.5, OSP 10 has 1.3.0 (!). This is a huge difference, we can probably risk rebasing if it solves problems.

Comment 9 Bob Fournier 2018-02-14 14:55:17 UTC
In OSP-9 it is recommended to use pxe_ipmitool per comments 7 and 8, or switch to the python-dracclient in OSP-10.  As there is has been response to the request for info in 4 months, closing this out.


Note You need to log in before you can comment on or make changes to this bug.