Hide Forgot
staging-ovirt driver: Nodes don't switch to managed upon import. Error: startswith first arg must be bytes or a tuple of bytes, not str Environment: python3-ovirt-engine-sdk4-4.2.9-4.el8ost.x86_64 Steps to reproduce: Deploy undercloud with the following in undercloud.conf: enabled_hardware_types = ipmi,staging-ovirt Attempt to import nodes using staging-ovirt for pm_type. Result: (undercloud) [stack@undercloud ~]$ openstack overcloud node import ./instackenv.json Waiting for messages on queue 'tripleo' with no timeout. [{'result': 'Node 9645afb1-aefa-4eb8-a24d-a12ec2a7eeb4 did not reach state "manageable", the state is "enroll", error: Failed to get power state for node 9645afb1-aefa-4eb8-a24d-a12ec2a7eeb4. Error: startswith first arg must be bytes or a tuple of bytes, not str'}, {'result': 'Node 60635e90-60fc-41d1-b789-dfd7c10d461f did not reach state "manageable", the state is "enroll", error: Failed to get power state for node 60635e90-60fc-41d1-b789-dfd7c10d461f. Error: startswith first arg must be bytes or a tuple of bytes, not str'}, {'result': 'Node 5c68ddca-52bf-4975-b4cf-98ab080be63d did not reach state "manageable", the state is "enroll", error: Failed to get power state for node 5c68ddca-52bf-4975-b4cf-98ab080be63d. Error: startswith first arg must be bytes or a tuple of bytes, not str'}, {'result': 'Node 6ce89c6a-e9fc-4cdf-b83c-a9266c067584 did not reach state "manageable", the state is "enroll", error: Failed to get power state for node 6ce89c6a-e9fc-4cdf-b83c-a9266c067584. Error: startswith first arg must be bytes or a tuple of bytes, not str'}] {'result': 'Failure caused by error in tasks: send_message\n\n send_message [task_ex_id=094c2bab-2265-47da-895f-24b3cb898a94] -> Workflow failed due to message status. Status:FAILED Message:({\'result\': \'Node 9645afb1-aefa-4eb8-a24d-a12ec2a7eeb4 did not reach state "manageable", the state is "enroll", error: Failed to get power state for node 9645afb1-aefa-4eb8-a24d-a12ec2a7eeb4. Error: startswith first arg must be bytes or a tuple of bytes, not str\'}, {\'result\': \'Node 60635e90-60fc-41d1-b789-dfd7c10d461f did not reach state "manageable", the state is "enroll", error: Failed to get power state for node 60635e90-60fc-41d1-b789-dfd7c10d461f. Error: startswith first arg must be bytes or a tuple of bytes, not str\'}, {\'result\': \'Node 5c68ddca-52bf-4975-b4cf-98ab080be63d did not reach state "manageable", the state is "enroll", error: Failed to get power state for node 5c68ddca-52bf-4975-b4cf-98ab080be63d. Error: startswith first arg must be bytes or a tuple of bytes, not str\'}, {\'result\': \'Node 6ce89c6a-e9fc-4cdf-b83c-a9266c067584 did not reach state "manageable", the state is "enroll", error: Failed to get power state for node 6ce89c6a-e9fc-4cdf-b83c-a9266c067584. Error: startswith first arg must be bytes or a tuple of bytes, not str\'})\n [wf_ex_id=14d41839-9b36-4f7b-971c-2f8eec85b689, idx=0]: Workflow failed due to message status. Status:FAILED Message:({\'result\': \'Node 9645afb1-aefa-4eb8-a24d-a12ec2a7eeb4 did not reach state "manageable", the state is "enroll", error: Failed to get power state for node 9645afb1-aefa-4eb8-a24d-a12ec2a7eeb4. Error: startswith first arg must be bytes or a tuple of bytes, not str\'}, {\'result\': \'Node 60635e90-60fc-41d1-b789-dfd7c10d461f did not reach state "manageable", the state is "enroll", error: Failed to get power state for node 60635e90-60fc-41d1-b789-dfd7c10d461f. Error: startswith first arg must be bytes or a tuple of bytes, not str\'}, {\'result\': \'Node 5c68ddca-52bf-4975-b4cf-98ab080be63d did not reach state "manageable", the state is "enroll", error: Failed to get power state for node 5c68ddca-52bf-4975-b4cf-98ab080be63d. Error: startswith first arg must be bytes or a tuple of bytes, not str\'}, {\'result\': \'Node 6ce89c6a-e9fc-4cdf-b83c-a9266c067584 did not reach state "manageable", the state is "enroll", error: Failed to get power state for node 6ce89c6a-e9fc-4cdf-b83c-a9266c067584. Error: startswith first arg must be bytes or a tuple of bytes, not str\'})\n', 'status': 'FAILED', 'message': [{'result': 'Node 9645afb1-aefa-4eb8-a24d-a12ec2a7eeb4 did not reach state "manageable", the state is "enroll", error: Failed to get power state for node 9645afb1-aefa-4eb8-a24d-a12ec2a7eeb4. Error: startswith first arg must be bytes or a tuple of bytes, not str'}, {'result': 'Node 60635e90-60fc-41d1-b789-dfd7c10d461f did not reach state "manageable", the state is "enroll", error: Failed to get power state for node 60635e90-60fc-41d1-b789-dfd7c10d461f. Error: startswith first arg must be bytes or a tuple of bytes, not str'}, {'result': 'Node 5c68ddca-52bf-4975-b4cf-98ab080be63d did not reach state "manageable", the state is "enroll", error: Failed to get power state for node 5c68ddca-52bf-4975-b4cf-98ab080be63d. Error: startswith first arg must be bytes or a tuple of bytes, not str'}, {'result': 'Node 6ce89c6a-e9fc-4cdf-b83c-a9266c067584 did not reach state "manageable", the state is "enroll", error: Failed to get power state for node 6ce89c6a-e9fc-4cdf-b83c-a9266c067584. Error: startswith first arg must be bytes or a tuple of bytes, not str'}]} Exception registering nodes: {'result': 'Failure caused by error in tasks: send_message\n\n send_message [task_ex_id=094c2bab-2265-47da-895f-24b3cb898a94] -> Workflow failed due to message status. Status:FAILED Message:({\'result\': \'Node 9645afb1-aefa-4eb8-a24d-a12ec2a7eeb4 did not reach state "manageable", the state is "enroll", error: Failed to get power state for node 9645afb1-aefa-4eb8-a24d-a12ec2a7eeb4. Error: startswith first arg must be bytes or a tuple of bytes, not str\'}, {\'result\': \'Node 60635e90-60fc-41d1-b789-dfd7c10d461f did not reach state "manageable", the state is "enroll", error: Failed to get power state for node 60635e90-60fc-41d1-b789-dfd7c10d461f. Error: startswith first arg must be bytes or a tuple of bytes, not str\'}, {\'result\': \'Node 5c68ddca-52bf-4975-b4cf-98ab080be63d did not reach state "manageable", the state is "enroll", error: Failed to get power state for node 5c68ddca-52bf-4975-b4cf-98ab080be63d. Error: startswith first arg must be bytes or a tuple of bytes, not str\'}, {\'result\': \'Node 6ce89c6a-e9fc-4cdf-b83c-a9266c067584 did not reach state "manageable", the state is "enroll", error: Failed to get power state for node 6ce89c6a-e9fc-4cdf-b83c-a9266c067584. Error: startswith first arg must be bytes or a tuple of bytes, not str\'})\n [wf_ex_id=14d41839-9b36-4f7b-971c-2f8eec85b689, idx=0]: Workflow failed due to message status. Status:FAILED Message:({\'result\': \'Node 9645afb1-aefa-4eb8-a24d-a12ec2a7eeb4 did not reach state "manageable", the state is "enroll", error: Failed to get power state for node 9645afb1-aefa-4eb8-a24d-a12ec2a7eeb4. Error: startswith first arg must be bytes or a tuple of bytes, not str\'}, {\'result\': \'Node 60635e90-60fc-41d1-b789-dfd7c10d461f did not reach state "manageable", the state is "enroll", error: Failed to get power state for node 60635e90-60fc-41d1-b789-dfd7c10d461f. Error: startswith first arg must be bytes or a tuple of bytes, not str\'}, {\'result\': \'Node 5c68ddca-52bf-4975-b4cf-98ab080be63d did not reach state "manageable", the state is "enroll", error: Failed to get power state for node 5c68ddca-52bf-4975-b4cf-98ab080be63d. Error: startswith first arg must be bytes or a tuple of bytes, not str\'}, {\'result\': \'Node 6ce89c6a-e9fc-4cdf-b83c-a9266c067584 did not reach state "manageable", the state is "enroll", error: Failed to get power state for node 6ce89c6a-e9fc-4cdf-b83c-a9266c067584. Error: startswith first arg must be bytes or a tuple of bytes, not str\'})\n', 'status': 'FAILED', 'message': [{'result': 'Node 9645afb1-aefa-4eb8-a24d-a12ec2a7eeb4 did not reach state "manageable", the state is "enroll", error: Failed to get power state for node 9645afb1-aefa-4eb8-a24d-a12ec2a7eeb4. Error: startswith first arg must be bytes or a tuple of bytes, not str'}, {'result': 'Node 60635e90-60fc-41d1-b789-dfd7c10d461f did not reach state "manageable", the state is "enroll", error: Failed to get power state for node 60635e90-60fc-41d1-b789-dfd7c10d461f. Error: startswith first arg must be bytes or a tuple of bytes, not str'}, {'result': 'Node 5c68ddca-52bf-4975-b4cf-98ab080be63d did not reach state "manageable", the state is "enroll", error: Failed to get power state for node 5c68ddca-52bf-4975-b4cf-98ab080be63d. Error: startswith first arg must be bytes or a tuple of bytes, not str'}, {'result': 'Node 6ce89c6a-e9fc-4cdf-b83c-a9266c067584 did not reach state "manageable", the state is "enroll", error: Failed to get power state for node 6ce89c6a-e9fc-4cdf-b83c-a9266c067584. Error: startswith first arg must be bytes or a tuple of bytes, not str'}]} (undercloud) [stack@undercloud ~]$
In ironic-conductor.log in ironic container: 2d4b37bebaebf74abf5b02 4dc9db25d37b4012a50cc9f260937181 - default default] Starting power credentials verification for node 6ce89c6a-e9fc-4cdf-b83c-a9266c067584 _do_node_verify /usr/lib/python3.6/site-packages/ironic/conductor/manager.py:1462 2019-07-01 02:30:46.850 8 ERROR ironic.conductor.manager [req-e7cb70c2-4b9d-4cec-b138-105cc151f75f 17340cb8a42d4b37bebaebf74abf5b02 4dc9db25d37b4012a50cc9f260937181 - default default] Failed to get power state for node 6ce89c6a-e9fc-4cdf-b83c-a9266c067584. Error: startswith first arg must be bytes or a tuple of bytes, not str 2019-07-01 02:30:46.850 8 DEBUG ironic.common.states [req-e7cb70c2-4b9d-4cec-b138-105cc151f75f 17340cb8a42d4b37bebaebf74abf5b02 4dc9db25d37b4012a50cc9f260937181 - default default] Exiting old state 'verifying' in response to event 'fail' on_exit /usr/lib/python3.6/site-packages/ironic/common/states.py:294
Created attachment 1586408 [details] ironic-conductor.log
For one of the failing nodes: | driver | staging-ovirt | | driver_info | {'deploy_kernel': '88198718-0b14-45a9-bf5e-a4fdb3683a3b', 'rescue_kernel': '88198718-0b14-45a9-bf5e-a4fdb3683a3b', 'deploy_ramdisk': '7612f712-34d5-44bb-8ab5-3130a6c5bd98', 'rescue_ramdisk': '7612f712-34d5-44bb-8ab5-3130a6c5bd98', 'ovirt_address': 'rhev.tlv.redhat.com', 'ovirt_username': 'sasha@rhev-tlv-ipa.eng.lab.tlv.redhat.com', 'ovirt_vm_name': 'achuzhoy-ctl01', 'ovirt_password': '******'} | | driver_internal_info | {}
Temporily removing blocker to that TR can be set.
I've tested overcloud deploy with the linked patch and it works to the extend that concerns ironic. There have been some Nova scheduler failure, but we thought these are irrelevant to the problem at hand.
Verified: Environment: openstack-ironic-staging-drivers-0.11.1-0.20190711161813.6f2589e.el8ost.noarch Successfully imported nodes using staging-ovirt driver and they switched to managed.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2019:2811