Description of problem: Having several nodes on OSP8 and trying to introspect them. This fails if there is no DISPLAY on the server hosting the VMs. Nodes were defined this way: . ./stackrc ; \ ironic node-create -n ${IRONIC_NODE} \ -d pxe_ssh \ -i ssh_address=${VBOX_HOST_IP} \ -i ssh_username=${VBOX_USER} \ -i ssh_virt_type=vbox \ -i ssh_password=\"${VBOX_USER_PWD}\" \ . ./stackrc ; \ ironic node-update ${IRONIC_UUID} add \ properties/cpus=${vm_slave_cpu_default} \ properties/memory_mb=${vm_slave_memory_default} \ properties/local_gb=62 \ properties/cpu_arch=x86_64 \ \"" 'openstack introspection bulk start' hung forever and In the logs I was able to notice these outputs: Error: Failed to execute command via SSH: LC_ALL=C /usr/bin/VBoxManage startvm osp-baremetal-9.", "updated_at": "2016-03-24T15:08:40+00:00", "maintenance_reason": null, "provision_state": "manageable", "clean_step": {}, "uuid": "71392099-f84c-449e-88a1-e3b1fd1af248", "console_enabled": false, "target_provision_state": null, "provision_updated_at": "2016-03-24T14:24:18+00:00", "power_state": "power on", "inspection_started_at": null, "inspection_finished_at": null, "maintenance": false, "driver": "pxe_ssh", "reservation": null, "properties": {"memory_mb": 4096, "cpu_arch": "x86_64", "local_gb": 62, "cpus": 2}, "instance_uuid": null, "name": "osp-baremetal-9", "driver_info": {"ssh_username": "raistlin", "ssh_virt_type": "vbox", "ssh_address": "10.0.128.244", "ssh_password": "******"}, "created_at": "2016-03-24T14:23:08+00:00", "ports": [{"href": "http://10.20.0.2:6385/v1/nodes/71392099-f84c-449e-88a1-e3b1fd1af248/ports", "rel": "self"}, {"href": "http://10.20.0.2:6385/nodes/71392099-f84c-449e-88a1-e3b1fd1af248/ports", "rel": "bookmark"}], "driver_internal_info": {}, "instance_info": {}}, {"target_power_state": null, "links": [{"href": "http://10.20.0.2:6385/v1/nodes/031d9a8e-f8fe-41bb-a41c-17be0519088d", "rel": "self"}, {"href" Sure enough, I was able to reproduce this at the CLI: $ VBoxManage startvm osp-baremetal-8 Waiting for VM "osp-baremetal-8" to power on... VBoxManage: error: The virtual machine 'osp-baremetal-8' has terminated unexpectedly during startup with exit code 1 (0x1) VBoxManage: error: Details: code NS_ERROR_FAILURE (0x80004005), component MachineWrap, interface IMachine The workaround is to invoked startvm with '--type headless': $ VBoxManage startvm --type headless osp-baremetal-8 Waiting for VM "osp-baremetal-8" to power on... VM "osp-baremetal-8" has been successfully started. Version-Release number of selected component (if applicable): How reproducible: 100% if none is logged at the console (no DISPLAY running) Additional info: stack@instack ~]$ ironic node-list +--------------------------------------+------------------+---------------+-------------+--------------------+-------------+ | UUID | Name | Instance UUID | Power State | Provisioning State | Maintenance | +--------------------------------------+------------------+---------------+-------------+--------------------+-------------+ | 4caf2c9d-1d04-4863-b4ae-810e560bcbcc | osp-baremetal-1 | None | power off | available | False | | 5f0c0f63-53fa-4f13-9d4c-4565eb8b75e4 | osp-baremetal-2 | None | power off | available | False | | 53bc95dd-bf1a-49ec-ae22-518edabfe019 | osp-baremetal-3 | None | power off | available | False | | 61065f76-0160-4ca4-873b-e3b6da944e5e | osp-baremetal-4 | None | power off | available | False | | 82b73be8-d1b2-49d0-811d-751c58722a0f | osp-baremetal-5 | None | power off | available | False | | c8cd6846-1b09-427e-95b4-0cb7a4aafc3e | osp-baremetal-6 | None | power off | available | False | | 30c63703-13d7-4b82-8c22-c83e972fb7cd | osp-baremetal-7 | None | power off | available | False | | 74a19eb4-1c90-4afb-8b8b-b5f51be9d9ef | osp-baremetal-8 | None | power off | available | False | | 71392099-f84c-449e-88a1-e3b1fd1af248 | osp-baremetal-9 | None | power off | available | False | | 031d9a8e-f8fe-41bb-a41c-17be0519088d | osp-baremetal-10 | None | power off | available | False | | 927b3514-6d6f-4d80-af00-47442c266871 | osp-baremetal-11 | None | power off | available | False | | 18e66816-c35b-4473-a4e9-df8d3c6f4fad | osp-baremetal-12 | None | power off | available | False | +--------------------------------------+------------------+---------------+-------------+--------------------+-------------+ [stack@instack ~]$ ironic node-show 74a19eb4-1c90-4afb-8b8b-b5f51be9d9ef +------------------------+----------------------------------------------------------------------+ | Property | Value | +------------------------+----------------------------------------------------------------------+ | target_power_state | None | | extra | {u'hardware_swift_object': u'extra_hardware-74a19eb4-1c90-4afb-8b8b- | | | b5f51be9d9ef'} | | last_error | None | | updated_at | 2016-03-24T16:35:37+00:00 | | maintenance_reason | None | | provision_state | available | | clean_step | {} | | uuid | 74a19eb4-1c90-4afb-8b8b-b5f51be9d9ef | | console_enabled | False | | target_provision_state | None | | provision_updated_at | 2016-03-24T15:11:34+00:00 | | maintenance | False | | inspection_started_at | None | | inspection_finished_at | None | | power_state | power off | | driver | pxe_ssh | | reservation | None | | properties | {u'memory_mb': u'4096', u'cpu_arch': u'x86_64', u'local_gb': u'62', | | | u'cpus': u'2'} | | instance_uuid | None | | name | osp-baremetal-8 | | driver_info | {u'ssh_username': u'raistlin', u'ssh_virt_type': u'vbox', | | | u'ssh_address': u'10.0.128.244', u'ssh_password': u'******'} | | created_at | 2016-03-24T14:22:57+00:00 | | driver_internal_info | {} | | chassis_uuid | | | instance_info | {} | +------------------------+----------------------------------------------------------------------+
# rpm -qf /lib/python2.7/site-packages/ironic/drivers/modules/ssh.py openstack-ironic-common-4.2.2-4.el7ost.noarch
I was testing this patch but it seems the .py code needs to be recompiled for that to work as I don't see my changes in the logs: # diff -c /lib/python2.7/site-packages/ironic/drivers/modules/ssh.py.orig /lib/python2.7/site-packages/ironic/drivers/modules/ssh.py *** /lib/python2.7/site-packages/ironic/drivers/modules/ssh.py.orig Fri Dec 4 16:39:19 2015 --- /lib/python2.7/site-packages/ironic/drivers/modules/ssh.py Thu Mar 24 12:35:13 2016 *************** *** 127,133 **** if virt_type == 'vbox': return { 'base_cmd': 'LC_ALL=C /usr/bin/VBoxManage', ! 'start_cmd': 'startvm {_NodeName_}', 'stop_cmd': 'controlvm {_NodeName_} poweroff', 'reboot_cmd': 'controlvm {_NodeName_} reset', 'list_all': "list vms|awk -F'\"' '{print $2}'", --- 127,133 ---- if virt_type == 'vbox': return { 'base_cmd': 'LC_ALL=C /usr/bin/VBoxManage', ! 'start_cmd': 'startvm --type headless {_NodeName_}', 'stop_cmd': 'controlvm {_NodeName_} poweroff', 'reboot_cmd': 'controlvm {_NodeName_} reset', 'list_all': "list vms|awk -F'\"' '{print $2}'",
After putting in the patch and recompile ssh.pyc, it works sucessfully: # grep -iA1 headless * ironic-conductor.log:2016-03-24 13:03:54.549 19778 DEBUG oslo_concurrency.processutils [-] Running cmd (SSH): LC_ALL=C /usr/bin/VBoxManage startvm --type headless osp-baremetal-1 ssh_execute /usr/lib/python2.7/site-packages/oslo_concurrency/processutils.py:357 ironic-conductor.log-2016-03-24 13:03:54.657 19778 DEBUG oslo_concurrency.processutils [req-29a9ab27-a643-40e8-a013-d31144000cf8 ] Result was 0 ssh_execute /usr/lib/python2.7/site-packages/oslo_concurrency/processutils.py:388 -- ironic-conductor.log:2016-03-24 13:04:02.689 19778 DEBUG oslo_concurrency.processutils [-] Running cmd (SSH): LC_ALL=C /usr/bin/VBoxManage startvm --type headless osp-baremetal-2 ssh_execute /usr/lib/python2.7/site-packages/oslo_concurrency/processutils.py:357 ironic-conductor.log-2016-03-24 13:04:02.817 19778 DEBUG oslo_concurrency.processutils [-] Result was 0 ssh_execute /usr/lib/python2.7/site-packages/oslo_concurrency/processutils.py:388 -- ironic-conductor.log:2016-03-24 13:04:09.035 19778 DEBUG oslo_concurrency.processutils [-] Running cmd (SSH): LC_ALL=C /usr/bin/VBoxManage startvm --type headless osp-baremetal-3 ssh_execute /usr/lib/python2.7/site-packages/oslo_concurrency/processutils.py:357 ironic-conductor.log-2016-03-24 13:04:09.216 19778 DEBUG ironic.drivers.modules.deploy_utils [req-a512e0e8-e840-4aa5-a8c6-a54a39b2441b ] Deploy boot mode is None for 82b73be8-d1b2-49d0-811d-751c58722a0f. get_boot_mode_for_deploy /usr/lib/python2.7/site-packages/ironic/drivers/modules/deploy_utils.py:1174 -- ironic-conductor.log:2016-03-24 13:04:20.421 19778 DEBUG oslo_concurrency.processutils [-] Running cmd (SSH): LC_ALL=C /usr/bin/VBoxManage startvm --type headless osp-baremetal-4 ssh_execute /usr/lib/python2.7/site-packages/oslo_concurrency/processutils.py:357 ironic-conductor.log-2016-03-24 13:04:20.546 19778 DEBUG oslo_concurrency.processutils [-] Result was 0 ssh_execute /usr/lib/python2.7/site-packages/oslo_concurrency/processutils.py:388 -- ironic-conductor.log:2016-03-24 13:04:37.084 19778 DEBUG oslo_concurrency.processutils [-] Running cmd (SSH): LC_ALL=C /usr/bin/VBoxManage startvm --type headless osp-baremetal-5 ssh_execute /usr/lib/python2.7/site-packages/oslo_concurrency/processutils.py:357 ironic-conductor.log-2016-03-24 13:04:37.284 19778 DEBUG oslo_concurrency.processutils [-] Result was 0 ssh_execute /usr/lib/python2.7/site-packages/oslo_concurrency/processutils.py:388 -- ironic-conductor.log:2016-03-24 13:04:41.713 19778 DEBUG oslo_concurrency.processutils [-] Running cmd (SSH): LC_ALL=C /usr/bin/VBoxManage startvm --type headless osp-baremetal-6 ssh_execute /usr/lib/python2.7/site-packages/oslo_concurrency/processutils.py:357 ironic-conductor.log-2016-03-24 13:04:41.895 19778 DEBUG oslo_concurrency.processutils [req-aa64523b-d3fd-4866-bacf-7d2b78556f73 ] Result was 0 ssh_execute /usr/lib/python2.7/site-packages/oslo_concurrency/processutils.py:388 --
Hi everyone, My final patch was accepted upstream: https://review.openstack.org/#/c/311278 I am using it on OSP8 to make use of VBox as an hypervisor on my RHEL6.8 boxes. Here's what I am doing on top of OSP8 to make it work: $ sudo patch -p0 < ssh.py.patch /usr/lib/python2.7/site-packages/ironic/drivers/modules/ssh.py $ sudo python -m compileall /usr/lib/python2.7/site-packages/ironic/drivers/modules/ssh.py I'm attaching the patch to this BZ. Thanks, Vincent
Created attachment 1176530 [details] Patch for OSP8
Hi Vicent, Thanks, here's the backported patch to OSP8: https://code.engineering.redhat.com/gerrit/#/c/78275/
Verified per comment #4 and comment #5
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHBA-2016-2709.html