Bug 1725503 - staging-ovirt driver: Nodes don't switch to managed upon import. Error: startswith first arg must be bytes or a tuple of bytes, not str
Summary: staging-ovirt driver: Nodes don't switch to managed upon import. Error: sta...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-ironic-staging-drivers
Version: 15.0 (Stein)
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: beta
: 15.0 (Stein)
Assignee: Ilya Etingof
QA Contact: Alexander Chuzhoy
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-06-30 23:35 UTC by Alexander Chuzhoy
Modified: 2019-09-26 10:53 UTC (History)
7 users (show)

Fixed In Version: openstack-ironic-staging-drivers-0.11.1-0.20190711161813.6f2589e.el8ost
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-09-21 11:23:46 UTC
Target Upstream Version:
ietingof: needinfo-
ietingof: needinfo-


Attachments (Terms of Use)
ironic-conductor.log (6.34 MB, text/plain)
2019-07-01 19:50 UTC, Bob Fournier
no flags Details


Links
System ID Private Priority Status Summary Last Updated
OpenStack gerrit 668876 0 None MERGED Fix pycurl args types in ovirt driver 2020-10-06 17:49:32 UTC
OpenStack gerrit 670273 0 None MERGED Fix pycurl args types in ovirt driver 2020-10-06 17:49:31 UTC
Red Hat Product Errata RHEA-2019:2811 0 None None None 2019-09-21 11:24:14 UTC

Description Alexander Chuzhoy 2019-06-30 23:35:17 UTC
staging-ovirt driver: Nodes don't switch to managed upon import.   Error: startswith first arg must be bytes or a tuple of bytes, not str


Environment:
python3-ovirt-engine-sdk4-4.2.9-4.el8ost.x86_64


Steps to reproduce:

Deploy undercloud with the following in undercloud.conf:
enabled_hardware_types = ipmi,staging-ovirt


Attempt to import nodes using staging-ovirt for pm_type.


Result:

(undercloud) [stack@undercloud ~]$ openstack overcloud node import ./instackenv.json 
Waiting for messages on queue 'tripleo' with no timeout.

[{'result': 'Node 9645afb1-aefa-4eb8-a24d-a12ec2a7eeb4 did not reach state "manageable", the state is "enroll", error: Failed to get power state for node 9645afb1-aefa-4eb8-a24d-a12ec2a7eeb4. Error: startswith first arg must be bytes or a tuple of bytes, not str'}, {'result': 'Node 60635e90-60fc-41d1-b789-dfd7c10d461f did not reach state "manageable", the state is "enroll", error: Failed to get power state for node 60635e90-60fc-41d1-b789-dfd7c10d461f. Error: startswith first arg must be bytes or a tuple of bytes, not str'}, {'result': 'Node 5c68ddca-52bf-4975-b4cf-98ab080be63d did not reach state "manageable", the state is "enroll", error: Failed to get power state for node 5c68ddca-52bf-4975-b4cf-98ab080be63d. Error: startswith first arg must be bytes or a tuple of bytes, not str'}, {'result': 'Node 6ce89c6a-e9fc-4cdf-b83c-a9266c067584 did not reach state "manageable", the state is "enroll", error: Failed to get power state for node 6ce89c6a-e9fc-4cdf-b83c-a9266c067584. Error: startswith first arg must be bytes or a tuple of bytes, not str'}]
{'result': 'Failure caused by error in tasks: send_message\n\n  send_message [task_ex_id=094c2bab-2265-47da-895f-24b3cb898a94] -> Workflow failed due to message status. Status:FAILED Message:({\'result\': \'Node 9645afb1-aefa-4eb8-a24d-a12ec2a7eeb4 did not reach state "manageable", the state is "enroll", error: Failed to get power state for node 9645afb1-aefa-4eb8-a24d-a12ec2a7eeb4. Error: startswith first arg must be bytes or a tuple of bytes, not str\'}, {\'result\': \'Node 60635e90-60fc-41d1-b789-dfd7c10d461f did not reach state "manageable", the state is "enroll", error: Failed to get power state for node 60635e90-60fc-41d1-b789-dfd7c10d461f. Error: startswith first arg must be bytes or a tuple of bytes, not str\'}, {\'result\': \'Node 5c68ddca-52bf-4975-b4cf-98ab080be63d did not reach state "manageable", the state is "enroll", error: Failed to get power state for node 5c68ddca-52bf-4975-b4cf-98ab080be63d. Error: startswith first arg must be bytes or a tuple of bytes, not str\'}, {\'result\': \'Node 6ce89c6a-e9fc-4cdf-b83c-a9266c067584 did not reach state "manageable", the state is "enroll", error: Failed to get power state for node 6ce89c6a-e9fc-4cdf-b83c-a9266c067584. Error: startswith first arg must be bytes or a tuple of bytes, not str\'})\n    [wf_ex_id=14d41839-9b36-4f7b-971c-2f8eec85b689, idx=0]: Workflow failed due to message status. Status:FAILED Message:({\'result\': \'Node 9645afb1-aefa-4eb8-a24d-a12ec2a7eeb4 did not reach state "manageable", the state is "enroll", error: Failed to get power state for node 9645afb1-aefa-4eb8-a24d-a12ec2a7eeb4. Error: startswith first arg must be bytes or a tuple of bytes, not str\'}, {\'result\': \'Node 60635e90-60fc-41d1-b789-dfd7c10d461f did not reach state "manageable", the state is "enroll", error: Failed to get power state for node 60635e90-60fc-41d1-b789-dfd7c10d461f. Error: startswith first arg must be bytes or a tuple of bytes, not str\'}, {\'result\': \'Node 5c68ddca-52bf-4975-b4cf-98ab080be63d did not reach state "manageable", the state is "enroll", error: Failed to get power state for node 5c68ddca-52bf-4975-b4cf-98ab080be63d. Error: startswith first arg must be bytes or a tuple of bytes, not str\'}, {\'result\': \'Node 6ce89c6a-e9fc-4cdf-b83c-a9266c067584 did not reach state "manageable", the state is "enroll", error: Failed to get power state for node 6ce89c6a-e9fc-4cdf-b83c-a9266c067584. Error: startswith first arg must be bytes or a tuple of bytes, not str\'})\n', 'status': 'FAILED', 'message': [{'result': 'Node 9645afb1-aefa-4eb8-a24d-a12ec2a7eeb4 did not reach state "manageable", the state is "enroll", error: Failed to get power state for node 9645afb1-aefa-4eb8-a24d-a12ec2a7eeb4. Error: startswith first arg must be bytes or a tuple of bytes, not str'}, {'result': 'Node 60635e90-60fc-41d1-b789-dfd7c10d461f did not reach state "manageable", the state is "enroll", error: Failed to get power state for node 60635e90-60fc-41d1-b789-dfd7c10d461f. Error: startswith first arg must be bytes or a tuple of bytes, not str'}, {'result': 'Node 5c68ddca-52bf-4975-b4cf-98ab080be63d did not reach state "manageable", the state is "enroll", error: Failed to get power state for node 5c68ddca-52bf-4975-b4cf-98ab080be63d. Error: startswith first arg must be bytes or a tuple of bytes, not str'}, {'result': 'Node 6ce89c6a-e9fc-4cdf-b83c-a9266c067584 did not reach state "manageable", the state is "enroll", error: Failed to get power state for node 6ce89c6a-e9fc-4cdf-b83c-a9266c067584. Error: startswith first arg must be bytes or a tuple of bytes, not str'}]}
Exception registering nodes: {'result': 'Failure caused by error in tasks: send_message\n\n  send_message [task_ex_id=094c2bab-2265-47da-895f-24b3cb898a94] -> Workflow failed due to message status. Status:FAILED Message:({\'result\': \'Node 9645afb1-aefa-4eb8-a24d-a12ec2a7eeb4 did not reach state "manageable", the state is "enroll", error: Failed to get power state for node 9645afb1-aefa-4eb8-a24d-a12ec2a7eeb4. Error: startswith first arg must be bytes or a tuple of bytes, not str\'}, {\'result\': \'Node 60635e90-60fc-41d1-b789-dfd7c10d461f did not reach state "manageable", the state is "enroll", error: Failed to get power state for node 60635e90-60fc-41d1-b789-dfd7c10d461f. Error: startswith first arg must be bytes or a tuple of bytes, not str\'}, {\'result\': \'Node 5c68ddca-52bf-4975-b4cf-98ab080be63d did not reach state "manageable", the state is "enroll", error: Failed to get power state for node 5c68ddca-52bf-4975-b4cf-98ab080be63d. Error: startswith first arg must be bytes or a tuple of bytes, not str\'}, {\'result\': \'Node 6ce89c6a-e9fc-4cdf-b83c-a9266c067584 did not reach state "manageable", the state is "enroll", error: Failed to get power state for node 6ce89c6a-e9fc-4cdf-b83c-a9266c067584. Error: startswith first arg must be bytes or a tuple of bytes, not str\'})\n    [wf_ex_id=14d41839-9b36-4f7b-971c-2f8eec85b689, idx=0]: Workflow failed due to message status. Status:FAILED Message:({\'result\': \'Node 9645afb1-aefa-4eb8-a24d-a12ec2a7eeb4 did not reach state "manageable", the state is "enroll", error: Failed to get power state for node 9645afb1-aefa-4eb8-a24d-a12ec2a7eeb4. Error: startswith first arg must be bytes or a tuple of bytes, not str\'}, {\'result\': \'Node 60635e90-60fc-41d1-b789-dfd7c10d461f did not reach state "manageable", the state is "enroll", error: Failed to get power state for node 60635e90-60fc-41d1-b789-dfd7c10d461f. Error: startswith first arg must be bytes or a tuple of bytes, not str\'}, {\'result\': \'Node 5c68ddca-52bf-4975-b4cf-98ab080be63d did not reach state "manageable", the state is "enroll", error: Failed to get power state for node 5c68ddca-52bf-4975-b4cf-98ab080be63d. Error: startswith first arg must be bytes or a tuple of bytes, not str\'}, {\'result\': \'Node 6ce89c6a-e9fc-4cdf-b83c-a9266c067584 did not reach state "manageable", the state is "enroll", error: Failed to get power state for node 6ce89c6a-e9fc-4cdf-b83c-a9266c067584. Error: startswith first arg must be bytes or a tuple of bytes, not str\'})\n', 'status': 'FAILED', 'message': [{'result': 'Node 9645afb1-aefa-4eb8-a24d-a12ec2a7eeb4 did not reach state "manageable", the state is "enroll", error: Failed to get power state for node 9645afb1-aefa-4eb8-a24d-a12ec2a7eeb4. Error: startswith first arg must be bytes or a tuple of bytes, not str'}, {'result': 'Node 60635e90-60fc-41d1-b789-dfd7c10d461f did not reach state "manageable", the state is "enroll", error: Failed to get power state for node 60635e90-60fc-41d1-b789-dfd7c10d461f. Error: startswith first arg must be bytes or a tuple of bytes, not str'}, {'result': 'Node 5c68ddca-52bf-4975-b4cf-98ab080be63d did not reach state "manageable", the state is "enroll", error: Failed to get power state for node 5c68ddca-52bf-4975-b4cf-98ab080be63d. Error: startswith first arg must be bytes or a tuple of bytes, not str'}, {'result': 'Node 6ce89c6a-e9fc-4cdf-b83c-a9266c067584 did not reach state "manageable", the state is "enroll", error: Failed to get power state for node 6ce89c6a-e9fc-4cdf-b83c-a9266c067584. Error: startswith first arg must be bytes or a tuple of bytes, not str'}]}
(undercloud) [stack@undercloud ~]$

Comment 1 Bob Fournier 2019-07-01 19:49:00 UTC
In ironic-conductor.log in ironic container:

2d4b37bebaebf74abf5b02 4dc9db25d37b4012a50cc9f260937181 - default default] Starting power credentials verification for node 6ce89c6a-e9fc-4cdf-b83c-a9266c067584 _do_node_verify /usr/lib/python3.6/site-packages/ironic/conductor/manager.py:1462
2019-07-01 02:30:46.850 8 ERROR ironic.conductor.manager [req-e7cb70c2-4b9d-4cec-b138-105cc151f75f 17340cb8a42d4b37bebaebf74abf5b02 4dc9db25d37b4012a50cc9f260937181 - default default] Failed to get power state for node 6ce89c6a-e9fc-4cdf-b83c-a9266c067584. Error: startswith first arg must be bytes or a tuple of bytes, not str
2019-07-01 02:30:46.850 8 DEBUG ironic.common.states [req-e7cb70c2-4b9d-4cec-b138-105cc151f75f 17340cb8a42d4b37bebaebf74abf5b02 4dc9db25d37b4012a50cc9f260937181 - default default] Exiting old state 'verifying' in response to event 'fail' on_exit /usr/lib/python3.6/site-packages/ironic/common/states.py:294

Comment 2 Bob Fournier 2019-07-01 19:50:15 UTC
Created attachment 1586408 [details]
ironic-conductor.log

Comment 3 Bob Fournier 2019-07-01 20:08:51 UTC
For one of the failing nodes:

| driver                 | staging-ovirt                                                                                                                                                                                                                                                                                                                                                                                                |
| driver_info            | {'deploy_kernel': '88198718-0b14-45a9-bf5e-a4fdb3683a3b', 'rescue_kernel': '88198718-0b14-45a9-bf5e-a4fdb3683a3b', 'deploy_ramdisk': '7612f712-34d5-44bb-8ab5-3130a6c5bd98', 'rescue_ramdisk': '7612f712-34d5-44bb-8ab5-3130a6c5bd98', 'ovirt_address': 'rhev.tlv.redhat.com', 'ovirt_username': 'sasha@rhev-tlv-ipa.eng.lab.tlv.redhat.com', 'ovirt_vm_name': 'achuzhoy-ctl01', 'ovirt_password': '******'} |
| driver_internal_info   | {}

Comment 4 Bob Fournier 2019-07-08 14:34:43 UTC
Temporily removing blocker to that TR can be set.

Comment 5 Ilya Etingof 2019-07-08 15:04:33 UTC
I've tested overcloud deploy with the linked patch and it works to the extend that concerns ironic. There have been some Nova scheduler failure, but we thought these are irrelevant to the problem at hand.

Comment 12 Alexander Chuzhoy 2019-07-16 17:31:21 UTC
Verified:

Environment:
openstack-ironic-staging-drivers-0.11.1-0.20190711161813.6f2589e.el8ost.noarch


Successfully imported nodes using staging-ovirt driver and they switched to managed.

Comment 16 errata-xmlrpc 2019-09-21 11:23:46 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2019:2811


Note You need to log in before you can comment on or make changes to this bug.