Bug 1321103 - pxe_ssh should use 'VBoxManage startvm --type headless' when ssh_virt_type is set to 'vbox'
Summary: pxe_ssh should use 'VBoxManage startvm --type headless' when ssh_virt_type is...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-ironic
Version: 8.0 (Liberty)
Hardware: Unspecified
OS: Unspecified
low
unspecified
Target Milestone: async
: 8.0 (Liberty)
Assignee: Lucas Alvares Gomes
QA Contact: Raviv Bar-Tal
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-03-24 16:45 UTC by Vincent S. Cojot
Modified: 2016-11-14 19:59 UTC (History)
3 users (show)

Fixed In Version: openstack-ironic-4.2.5-3.el7ost
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2016-11-14 19:59:02 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
Patch for OSP8 (4.04 KB, patch)
2016-07-05 15:15 UTC, Vincent S. Cojot
no flags Details | Diff


Links
System ID Private Priority Status Summary Last Updated
Launchpad 1561624 0 None None None 2016-03-24 16:45:23 UTC
Red Hat Product Errata RHBA-2016:2709 0 normal SHIPPED_LIVE openstack-ironic bug fix advisory 2016-11-15 00:55:04 UTC

Description Vincent S. Cojot 2016-03-24 16:45:23 UTC
Description of problem:

Having several nodes on OSP8 and trying to introspect them.
This fails if there is no DISPLAY on the server hosting the VMs.

Nodes were defined this way:

                . ./stackrc ; \
                ironic node-create -n ${IRONIC_NODE} \
                -d pxe_ssh \
                -i ssh_address=${VBOX_HOST_IP} \
                -i ssh_username=${VBOX_USER} \
                -i ssh_virt_type=vbox \
                -i ssh_password=\"${VBOX_USER_PWD}\" \

                . ./stackrc ; \
                ironic node-update ${IRONIC_UUID} add \
                properties/cpus=${vm_slave_cpu_default} \
                properties/memory_mb=${vm_slave_memory_default} \
                properties/local_gb=62 \
                properties/cpu_arch=x86_64 \
                \""

'openstack introspection bulk start' hung forever and In the logs I was able to notice these outputs:
 Error: Failed to execute command via SSH: LC_ALL=C /usr/bin/VBoxManage startvm osp-baremetal-9.", "updated_at": "2016-03-24T15:08:40+00:00", "maintenance_reason": null, "provision_state": "manageable", "clean_step": {}, "uuid": "71392099-f84c-449e-88a1-e3b1fd1af248", "console_enabled": false, "target_provision_state": null, "provision_updated_at": "2016-03-24T14:24:18+00:00", "power_state": "power on", "inspection_started_at": null, "inspection_finished_at": null, "maintenance": false, "driver": "pxe_ssh", "reservation": null, "properties": {"memory_mb": 4096, "cpu_arch": "x86_64", "local_gb": 62, "cpus": 2}, "instance_uuid": null, "name": "osp-baremetal-9", "driver_info": {"ssh_username": "raistlin", "ssh_virt_type": "vbox", "ssh_address": "10.0.128.244", "ssh_password": "******"}, "created_at": "2016-03-24T14:23:08+00:00", "ports": [{"href": "http://10.20.0.2:6385/v1/nodes/71392099-f84c-449e-88a1-e3b1fd1af248/ports", "rel": "self"}, {"href": "http://10.20.0.2:6385/nodes/71392099-f84c-449e-88a1-e3b1fd1af248/ports", "rel": "bookmark"}], "driver_internal_info": {}, "instance_info": {}}, {"target_power_state": null, "links": [{"href": "http://10.20.0.2:6385/v1/nodes/031d9a8e-f8fe-41bb-a41c-17be0519088d", "rel": "self"}, {"href"

Sure enough, I was able to reproduce this at the CLI:
$ VBoxManage startvm osp-baremetal-8
Waiting for VM "osp-baremetal-8" to power on...
VBoxManage: error: The virtual machine 'osp-baremetal-8' has terminated unexpectedly during startup with exit code 1 (0x1)
VBoxManage: error: Details: code NS_ERROR_FAILURE (0x80004005), component MachineWrap, interface IMachine

The workaround is to invoked startvm with '--type headless':
$ VBoxManage startvm --type headless osp-baremetal-8
Waiting for VM "osp-baremetal-8" to power on...
VM "osp-baremetal-8" has been successfully started.
Version-Release number of selected component (if applicable):


How reproducible:

100% if none is logged at the console (no DISPLAY running)


Additional info:
stack@instack ~]$ ironic node-list
+--------------------------------------+------------------+---------------+-------------+--------------------+-------------+
| UUID | Name | Instance UUID | Power State | Provisioning State | Maintenance |
+--------------------------------------+------------------+---------------+-------------+--------------------+-------------+
| 4caf2c9d-1d04-4863-b4ae-810e560bcbcc | osp-baremetal-1 | None | power off | available | False |
| 5f0c0f63-53fa-4f13-9d4c-4565eb8b75e4 | osp-baremetal-2 | None | power off | available | False |
| 53bc95dd-bf1a-49ec-ae22-518edabfe019 | osp-baremetal-3 | None | power off | available | False |
| 61065f76-0160-4ca4-873b-e3b6da944e5e | osp-baremetal-4 | None | power off | available | False |
| 82b73be8-d1b2-49d0-811d-751c58722a0f | osp-baremetal-5 | None | power off | available | False |
| c8cd6846-1b09-427e-95b4-0cb7a4aafc3e | osp-baremetal-6 | None | power off | available | False |
| 30c63703-13d7-4b82-8c22-c83e972fb7cd | osp-baremetal-7 | None | power off | available | False |
| 74a19eb4-1c90-4afb-8b8b-b5f51be9d9ef | osp-baremetal-8 | None | power off | available | False |
| 71392099-f84c-449e-88a1-e3b1fd1af248 | osp-baremetal-9 | None | power off | available | False |
| 031d9a8e-f8fe-41bb-a41c-17be0519088d | osp-baremetal-10 | None | power off | available | False |
| 927b3514-6d6f-4d80-af00-47442c266871 | osp-baremetal-11 | None | power off | available | False |
| 18e66816-c35b-4473-a4e9-df8d3c6f4fad | osp-baremetal-12 | None | power off | available | False |
+--------------------------------------+------------------+---------------+-------------+--------------------+-------------+
[stack@instack ~]$ ironic node-show 74a19eb4-1c90-4afb-8b8b-b5f51be9d9ef
+------------------------+----------------------------------------------------------------------+
| Property | Value |
+------------------------+----------------------------------------------------------------------+
| target_power_state | None |
| extra | {u'hardware_swift_object': u'extra_hardware-74a19eb4-1c90-4afb-8b8b- |
| | b5f51be9d9ef'} |
| last_error | None |
| updated_at | 2016-03-24T16:35:37+00:00 |
| maintenance_reason | None |
| provision_state | available |
| clean_step | {} |
| uuid | 74a19eb4-1c90-4afb-8b8b-b5f51be9d9ef |
| console_enabled | False |
| target_provision_state | None |
| provision_updated_at | 2016-03-24T15:11:34+00:00 |
| maintenance | False |
| inspection_started_at | None |
| inspection_finished_at | None |
| power_state | power off |
| driver | pxe_ssh |
| reservation | None |
| properties | {u'memory_mb': u'4096', u'cpu_arch': u'x86_64', u'local_gb': u'62', |
| | u'cpus': u'2'} |
| instance_uuid | None |
| name | osp-baremetal-8 |
| driver_info | {u'ssh_username': u'raistlin', u'ssh_virt_type': u'vbox', |
| | u'ssh_address': u'10.0.128.244', u'ssh_password': u'******'} |
| created_at | 2016-03-24T14:22:57+00:00 |
| driver_internal_info | {} |
| chassis_uuid | |
| instance_info | {} |
+------------------------+----------------------------------------------------------------------+

Comment 2 Vincent S. Cojot 2016-03-24 16:46:40 UTC
# rpm -qf /lib/python2.7/site-packages/ironic/drivers/modules/ssh.py
openstack-ironic-common-4.2.2-4.el7ost.noarch

Comment 3 Vincent S. Cojot 2016-03-24 16:51:32 UTC
I was testing this patch but it seems the .py code needs to be recompiled for that to work as I don't see my changes in the logs:

# diff -c /lib/python2.7/site-packages/ironic/drivers/modules/ssh.py.orig /lib/python2.7/site-packages/ironic/drivers/modules/ssh.py
*** /lib/python2.7/site-packages/ironic/drivers/modules/ssh.py.orig Fri Dec 4 16:39:19 2015
--- /lib/python2.7/site-packages/ironic/drivers/modules/ssh.py Thu Mar 24 12:35:13 2016
***************
*** 127,133 ****
      if virt_type == 'vbox':
          return {
              'base_cmd': 'LC_ALL=C /usr/bin/VBoxManage',
! 'start_cmd': 'startvm {_NodeName_}',
              'stop_cmd': 'controlvm {_NodeName_} poweroff',
              'reboot_cmd': 'controlvm {_NodeName_} reset',
              'list_all': "list vms|awk -F'\"' '{print $2}'",
--- 127,133 ----
      if virt_type == 'vbox':
          return {
              'base_cmd': 'LC_ALL=C /usr/bin/VBoxManage',
! 'start_cmd': 'startvm --type headless {_NodeName_}',
              'stop_cmd': 'controlvm {_NodeName_} poweroff',
              'reboot_cmd': 'controlvm {_NodeName_} reset',
              'list_all': "list vms|awk -F'\"' '{print $2}'",

Comment 4 Vincent S. Cojot 2016-03-24 17:11:54 UTC
After putting in the patch and recompile ssh.pyc, it works sucessfully:

# grep -iA1 headless *
ironic-conductor.log:2016-03-24 13:03:54.549 19778 DEBUG oslo_concurrency.processutils [-] Running cmd (SSH): LC_ALL=C /usr/bin/VBoxManage startvm --type headless osp-baremetal-1 ssh_execute /usr/lib/python2.7/site-packages/oslo_concurrency/processutils.py:357
ironic-conductor.log-2016-03-24 13:03:54.657 19778 DEBUG oslo_concurrency.processutils [req-29a9ab27-a643-40e8-a013-d31144000cf8 ] Result was 0 ssh_execute /usr/lib/python2.7/site-packages/oslo_concurrency/processutils.py:388
--
ironic-conductor.log:2016-03-24 13:04:02.689 19778 DEBUG oslo_concurrency.processutils [-] Running cmd (SSH): LC_ALL=C /usr/bin/VBoxManage startvm --type headless osp-baremetal-2 ssh_execute /usr/lib/python2.7/site-packages/oslo_concurrency/processutils.py:357
ironic-conductor.log-2016-03-24 13:04:02.817 19778 DEBUG oslo_concurrency.processutils [-] Result was 0 ssh_execute /usr/lib/python2.7/site-packages/oslo_concurrency/processutils.py:388
--
ironic-conductor.log:2016-03-24 13:04:09.035 19778 DEBUG oslo_concurrency.processutils [-] Running cmd (SSH): LC_ALL=C /usr/bin/VBoxManage startvm --type headless osp-baremetal-3 ssh_execute /usr/lib/python2.7/site-packages/oslo_concurrency/processutils.py:357
ironic-conductor.log-2016-03-24 13:04:09.216 19778 DEBUG ironic.drivers.modules.deploy_utils [req-a512e0e8-e840-4aa5-a8c6-a54a39b2441b ] Deploy boot mode is None for 82b73be8-d1b2-49d0-811d-751c58722a0f. get_boot_mode_for_deploy /usr/lib/python2.7/site-packages/ironic/drivers/modules/deploy_utils.py:1174
--
ironic-conductor.log:2016-03-24 13:04:20.421 19778 DEBUG oslo_concurrency.processutils [-] Running cmd (SSH): LC_ALL=C /usr/bin/VBoxManage startvm --type headless osp-baremetal-4 ssh_execute /usr/lib/python2.7/site-packages/oslo_concurrency/processutils.py:357
ironic-conductor.log-2016-03-24 13:04:20.546 19778 DEBUG oslo_concurrency.processutils [-] Result was 0 ssh_execute /usr/lib/python2.7/site-packages/oslo_concurrency/processutils.py:388
--
ironic-conductor.log:2016-03-24 13:04:37.084 19778 DEBUG oslo_concurrency.processutils [-] Running cmd (SSH): LC_ALL=C /usr/bin/VBoxManage startvm --type headless osp-baremetal-5 ssh_execute /usr/lib/python2.7/site-packages/oslo_concurrency/processutils.py:357
ironic-conductor.log-2016-03-24 13:04:37.284 19778 DEBUG oslo_concurrency.processutils [-] Result was 0 ssh_execute /usr/lib/python2.7/site-packages/oslo_concurrency/processutils.py:388
--
ironic-conductor.log:2016-03-24 13:04:41.713 19778 DEBUG oslo_concurrency.processutils [-] Running cmd (SSH): LC_ALL=C /usr/bin/VBoxManage startvm --type headless osp-baremetal-6 ssh_execute /usr/lib/python2.7/site-packages/oslo_concurrency/processutils.py:357
ironic-conductor.log-2016-03-24 13:04:41.895 19778 DEBUG oslo_concurrency.processutils [req-aa64523b-d3fd-4866-bacf-7d2b78556f73 ] Result was 0 ssh_execute /usr/lib/python2.7/site-packages/oslo_concurrency/processutils.py:388
--

Comment 5 Vincent S. Cojot 2016-07-05 15:14:23 UTC
Hi everyone,
My final patch was accepted upstream:
https://review.openstack.org/#/c/311278
I am using it on OSP8 to make use of VBox as an hypervisor on my RHEL6.8 boxes.

Here's what I am doing on top of OSP8 to make it work:

$ sudo patch -p0 < ssh.py.patch   /usr/lib/python2.7/site-packages/ironic/drivers/modules/ssh.py
$ sudo python -m compileall /usr/lib/python2.7/site-packages/ironic/drivers/modules/ssh.py

I'm attaching the patch to this BZ.

Thanks,

Vincent

Comment 6 Vincent S. Cojot 2016-07-05 15:15:29 UTC
Created attachment 1176530 [details]
Patch for OSP8

Comment 7 Lucas Alvares Gomes 2016-07-05 15:31:08 UTC
Hi Vicent,

Thanks, here's the backported patch to OSP8: https://code.engineering.redhat.com/gerrit/#/c/78275/

Comment 11 Lon Hohberger 2016-11-07 21:01:15 UTC
Verified per comment #4 and comment #5

Comment 13 errata-xmlrpc 2016-11-14 19:59:02 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2016-2709.html


Note You need to log in before you can comment on or make changes to this bug.