Bug 1265194 - [ppc64le] cannot add ppc64le host into cluster / vdsm-network.service: control process exited, code=exited status=1
[ppc64le] cannot add ppc64le host into cluster / vdsm-network.service: contro...
Status: CLOSED DUPLICATE of bug 1264667
Product: Red Hat Enterprise Virtualization Manager
Classification: Red Hat
Component: vdsm (Show other bugs)
3.6.0
ppc64le Linux
high Severity urgent
: ---
: 3.6.0
Assigned To: Petr Horáček
Aharon Canan
network
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2015-09-22 07:07 EDT by Jiri Belka
Modified: 2016-02-10 14:46 EST (History)
14 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2015-10-07 13:07:53 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: Network
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
logs from ppc64le host (61.90 KB, application/x-gzip)
2015-09-22 07:07 EDT, Jiri Belka
no flags Details

  None (edit)
Description Jiri Belka 2015-09-22 07:07:27 EDT
Created attachment 1075775 [details]
logs from ppc64le host

Description of problem:

cannot add ppc64le host into cluster / vdsm-network.service: control process exited, code=exited status=1.

~~~
Failed to install Host ibm-p8-rhevm-hv-02. Failed to execute stage 'Closing up': Command '/bin/systemctl' failed to execute.
~~~

~~~
2015-09-22 06:33:43 DEBUG otopi.plugins.otopi.services.systemd plugin.execute:940 execute-output: ('/bin/systemctl', 'start', 'vdsmd.service') stderr:
A dependency job for vdsmd.service failed. See 'journalctl -xe' for details.

2015-09-22 06:33:43 DEBUG otopi.context context._executeMethod:155 method exception
Traceback (most recent call last):
  File "/tmp/ovirt-5nF8L2TvA1/pythonlib/otopi/context.py", line 145, in _executeMethod
    method['method']()
  File "/tmp/ovirt-5nF8L2TvA1/otopi-plugins/ovirt-host-deploy/vdsm/packages.py", line 224, in _start
    self.services.state('vdsmd', True)
  File "/tmp/ovirt-5nF8L2TvA1/otopi-plugins/otopi/services/systemd.py", line 141, in state
    'start' if state else 'stop'
  File "/tmp/ovirt-5nF8L2TvA1/otopi-plugins/otopi/services/systemd.py", line 80, in _executeServiceCommand
    raiseOnError=raiseOnError
  File "/tmp/ovirt-5nF8L2TvA1/pythonlib/otopi/plugin.py", line 945, in execute
    command=args[0],
RuntimeError: Command '/bin/systemctl' failed to execute
~~~

~~~
-- Logs begin at Tue 2015-09-22 04:58:13 EDT, end at Tue 2015-09-22 07:00:01 EDT. --
Sep 22 06:33:43 ibm-p8-rhevm-hv-02.klab.eng.bos.redhat.com systemd[1]: Starting Virtual Desktop Server Manager network restoration...
-- Subject: Unit vdsm-network.service has begun start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit vdsm-network.service has begun starting up.
Sep 22 06:33:43 ibm-p8-rhevm-hv-02.klab.eng.bos.redhat.com systemd[1]: vdsm-network.service: control process exited, code=exited status=1
Sep 22 06:33:43 ibm-p8-rhevm-hv-02.klab.eng.bos.redhat.com systemd[1]: Failed to start Virtual Desktop Server Manager network restoration.
-- Subject: Unit vdsm-network.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit vdsm-network.service has failed.
-- 
~~~

Version-Release number of selected component (if applicable):
sanlock-3.2.4-1.el7.ppc64le
mom-0.5.0-1.el7ev.noarch
vdsm-4.17.7-1.el7ev.noarch
libvirt-python-1.2.17-2.el7.ppc64le
rhevm-backend-3.6.0-0.16.master.el6.noarch

How reproducible:
100%

Steps to Reproduce:
1. install rhel7.2 with following repos:

Repo-baseurl : http://download.englab.brq.redhat.com/pub/rhel/rel-eng/RHEL-7.2-20150917.0/compose/Server/ppc64le/os/
Repo-baseurl : http://download.englab.brq.redhat.com/pub/rhel/rel-eng/RHEL-7.2-20150917.0/compose/Server-optional/ppc64le/os/
Repo-baseurl : http://download.lab.bos.redhat.com/rel-eng/repos/rhevh-rhel-7.2-candidate/ppc64le/
Repo-baseurl : http://10.34.63.204/pub/mirror/rhevm/3.6/3.6.0-13/el7Server
Repo-baseurl : http://download.lab.bos.redhat.com/rel-eng/repos/rhel-7.2-candidate/ppc64le/
^^^ last one needed as there's no sanlock etc in rhevh-rhel-7.2-candidate

2. yum install vdsm
3. add into engine via admin portal

Actual results:
failure probably because of vdsm-network

Expected results:
should work

Additional info:
for petitboot

  Device: (*) Specify paths/URLs manually
  Kernel: http://download.eng.bos.redhat.com/rel-eng/RHEL-7.2-20150917.0/compose/Server/ppc64le/os/ppc/ppc64/vmlinuz
  Initrd: http://download.eng.bos.redhat.com/rel-eng/RHEL-7.2-20150917.0/compose/Server/ppc64le/os/ppc/ppc64/initrd.img
  Device tree: <empty>
  Boot arguments: ro ip=dhcp inst.text inst.headless inst.repo=http://download.eng.bos.redhat.com/rel-eng/RHEL-7.2-20150917.0/compose/Server/ppc64le/os/ inst.selinux=0
Comment 1 Eko 2015-09-23 03:59:04 EDT
we can find the same issue when install vdsm on RHEL-7.2-20150917.0.
it will block the virt-who vdsm mode testing.
Comment 2 Petr Horáček 2015-09-25 07:49:54 EDT
This bug could related to [1] which occurs on x86_64
[1] https://bugzilla.redhat.com/show_bug.cgi?format=multiple&id=1264667

upgrade.log contains:
MainThread::ERROR::2015-09-22 06:33:43,723::upgrade::94::upgrade::(apply_upgrade) Failed to run upgrade-unified-persistence
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/vdsm/tool/upgrade.py", line 92, in apply_upgrade
    upgrade.run(ns, args)
  File "/usr/lib/python2.7/site-packages/vdsm/tool/unified_persistence.py", line 193, in run
    run()
  File "/usr/lib/python2.7/site-packages/vdsm/tool/unified_persistence.py", line 44, in run
    networks, bondings = _getNetInfo()
  File "/usr/lib/python2.7/site-packages/vdsm/tool/unified_persistence.py", line 131, in _getNetInfo
    networks = _processNetworks(netinfo)
  File "/usr/lib/python2.7/site-packages/vdsm/tool/unified_persistence.py", line 53, in _processNetworks
    defaultGateway = getDefaultGateway()
  File "/usr/lib/python2.7/site-packages/vdsm/netinfo.py", line 319, in getDefaultGateway
    output = routeShowGateways('main')
  File "/usr/lib/python2.7/site-packages/vdsm/ipwrapper.py", line 537, in routeShowGateways
    return _execCmd(command)
  File "/usr/lib/python2.7/site-packages/vdsm/ipwrapper.py", line 526, in _execCmd
    returnCode, output, error = execCmd(command)
  File "/usr/lib/python2.7/site-packages/vdsm/utils.py", line 664, in execCmd
    deathSignal=deathSignal, childUmask=childUmask)
  File "/usr/lib64/python2.7/site-packages/cpopen/__init__.py", line 52, in __init__
    stderr=stderr)
  File "/usr/lib64/python2.7/subprocess.py", line 712, in __init__
    restore_sigpipe=restore_sigpipe)
TypeError: _execute_child_v276() got an unexpected keyword argument 'restore_sigpipe'
Comment 3 Petr Horáček 2015-09-25 07:59:25 EDT
Jiri, which version of cpopen was used? It's possible, that [1] can fix it (or broke it).

[1] https://gerrit.ovirt.org/#/c/46300/9
Comment 4 Eko 2015-09-29 04:56:12 EDT
it's a duplicated issue with https://bugzilla.redhat.com/show_bug.cgi?id=1264667

According to Oved Ourfali's comment, I downgrade the python package from 2.7.5-33 to 2.7.5-32, vdsm-tool configure --force can be executed successfully, vdsm also can be restarted normally.

so maybe it's not a ovirt issue, because we didn't install any ovirt packages, we just prepare a RHEL-7.2-20150917.0-Server os, and then add http://bob.eng.lab.tlv.redhat.com/builds/vt16/el7/ as the yum repo for installing vdsm packages. it's also not a python-cpopen issue, because this package is the same version in RHEL-7.2-20150910.2 and RHEL-7.2-20150917.0. 

RHEL-7.2-20150910.2 [PASS]  
  => python-2.7.5-32  
  => python-cpopen-1.3-5
  => vdsm-4.16.21-1.el7ev

RHEL-7.2-20150917.0 [FAILED]
  => python-2.7.5-33
  => python-cpopen-1.3-5
  => vdsm-4.16.21-1.el7ev
Comment 5 Yaniv Lavi 2015-10-07 05:57:37 EDT
Can you please try to recreate with the updated cpopen package and python version? Please close as duplicate if this works.
Comment 6 Jiri Belka 2015-10-07 06:11:50 EDT
Please see #5 as ppc64le hosts are under your control for now. Thx.
Comment 7 Artyom 2015-10-07 12:12:26 EDT
I not encounter such problem with:
vdsm-4.17.8-1.el7ev.noarch
python-cpopen-1.4-0.el7ev.ppc64le
python-2.7.5-33.el7.ppc64le
Comment 8 Petr Horáček 2015-10-07 13:07:53 EDT
Closing as a duplicate.

*** This bug has been marked as a duplicate of bug 1264667 ***

Note You need to log in before you can comment on or make changes to this bug.