Bug 1256212 - broken channel of rhel-7.2 (device-mapper-multipath, policycoreutils)
broken channel of rhel-7.2 (device-mapper-multipath, policycoreutils)
Status: CLOSED NOTABUG
Product: Red Hat Enterprise Virtualization Manager
Classification: Red Hat
Component: distribution (Show other bugs)
3.6.0
x86_64 Linux
unspecified Severity urgent
: ---
: ---
Assigned To: Joey Boggs
releng
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2015-08-24 00:52 EDT by Nikolai Sednev
Modified: 2016-02-10 15:12 EST (History)
9 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2015-09-01 07:21:59 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: Rel-Eng
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
engine.logs (322.13 KB, application/x-gzip)
2015-08-24 00:52 EDT, Nikolai Sednev
no flags Details
engine_sosreport (7.23 MB, application/x-xz)
2015-08-24 00:53 EDT, Nikolai Sednev
no flags Details
sosreport_from_host (5.70 MB, application/x-xz)
2015-08-24 00:56 EDT, Nikolai Sednev
no flags Details

  None (edit)
Description Nikolai Sednev 2015-08-24 00:52:17 EDT
Created attachment 1066199 [details]
engine.logs

Description of problem:
Host installation failed Command returned failure code 1 during SSH session 


2015-08-24 07:30:13,350 ERROR [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (org.ovirt.thread.pool-7-thread-30) [342f74ca] Error during host alma02.
qa.lab.tlv.redhat.com install: java.io.IOException: Command returned failure code 1 during SSH session 'root@alma02.qa.lab.tlv.redhat.com'
        at org.ovirt.engine.core.uutils.ssh.SSHClient.executeCommand(SSHClient.java:527) [uutils.jar:]
        at org.ovirt.engine.core.uutils.ssh.SSHDialog.executeCommand(SSHDialog.java:312) [uutils.jar:]
        at org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase.execute(VdsDeployBase.java:567) [bll.jar:]
        at org.ovirt.engine.core.bll.hostdeploy.InstallVdsInternalCommand.installHost(InstallVdsInternalCommand.java:180) [bll.jar:]
        at org.ovirt.engine.core.bll.hostdeploy.InstallVdsInternalCommand.executeCommand(InstallVdsInternalCommand.java:93) [bll.jar:]
        at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1211) [bll.jar:]
        at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1355) [bll.jar:]
        at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1979) [bll.jar:]
        at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) [utils.jar:]
        at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) [utils.jar:]
        at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1392) [bll.jar:]
        at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:374) [bll.jar:]
        at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:475) [bll.jar:]
        at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:457) [bll.jar:]
        at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:667) [bll.jar:]
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.8.0_51]
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) [rt.jar:1.8.0_51]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_51]
        at java.lang.reflect.Method.invoke(Method.java:497) [rt.jar:1.8.0_51]
        at org.jboss.as.ee.component.ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptor.java:52) [jboss-as-ee.jar:7.5.3.
Final-redhat-2]
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1]
        at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) [jboss-invocation.jar:1.1.2.Final-redhat-1]
        at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:63) [jboss-as-ee.jar:7.5.3.Final-r
edhat-2]
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1]
        at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:374) [jboss-invocation.jar:1.1.2.Final-redhat-1]
        at org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.delegateInterception(Jsr299BindingsInterceptor.java:74) [jboss-as-weld.jar:7.5.3.Final-redhat-2]
        at org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.doMethodInterception(Jsr299BindingsInterceptor.java:84) [jboss-as-weld.jar:7.5.3.Final-redhat-2]
        at org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.processInvocation(Jsr299BindingsInterceptor.java:97) [jboss-as-weld.jar:7.5.3.Final-redhat-2]
        at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:63) [jboss-as-ee.jar:7.5.3.Final-redhat-2]
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1]



        at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation.jar:1.1.2.Final-redhat-1]
        at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:185) [jboss-as-ee.jar:7.5.3.Final-redhat-2]
        at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:185) [jboss-as-ee.jar:7.5.3.Final-redhat-2]
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1]
        at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation.jar:1.1.2.Final-redhat-1]
        at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:73) [jboss-as-ee.jar:7.5.3.Final-redhat-2]
        at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view3.runInternalAction(Unknown Source) [bll.jar:]
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.8.0_51]
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) [rt.jar:1.8.0_51]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_51]
        at java.lang.reflect.Method.invoke(Method.java:497) [rt.jar:1.8.0_51]
        at org.jboss.weld.util.reflection.SecureReflections$13.work(SecureReflections.java:267) [weld-core.jar:1.1.30.Final-redhat-1]
        at org.jboss.weld.util.reflection.SecureReflectionAccess.run(SecureReflectionAccess.java:52) [weld-core.jar:1.1.30.Final-redhat-1]
        at org.jboss.weld.util.reflection.SecureReflectionAccess.runAsInvocation(SecureReflectionAccess.java:137) [weld-core.jar:1.1.30.Final-redhat-1]
        at org.jboss.weld.util.reflection.SecureReflections.invoke(SecureReflections.java:263) [weld-core.jar:1.1.30.Final-redhat-1]
        at org.jboss.weld.bean.proxy.EnterpriseBeanProxyMethodHandler.invoke(EnterpriseBeanProxyMethodHandler.java:115) [weld-core.jar:1.1.30.Final-redhat-1]
        at org.jboss.weld.bean.proxy.EnterpriseTargetBeanInstance.invoke(EnterpriseTargetBeanInstance.java:56) [weld-core.jar:1.1.30.Final-redhat-1]
        at org.jboss.weld.bean.proxy.ProxyMethodHandler.invoke(ProxyMethodHandler.java:105) [weld-core.jar:1.1.30.Final-redhat-1]
        at org.ovirt.engine.core.bll.BackendCommandObjectsHandler$BackendInternal$BackendLocal$142795634$Proxy$_$$_Weld$Proxy$.runInternalAction(BackendCommandObjectsHandler$BackendInternal$BackendLocal$142795634$Proxy$_$$_Weld$Proxy$.java) [bll.jar:]
        at org.ovirt.engine.core.bll.CommandBase.runInternalAction(CommandBase.java:2342) [bll.jar:]
        at org.ovirt.engine.core.bll.hostdeploy.AddVdsCommand.access$800(AddVdsCommand.java:67) [bll.jar:]
        at org.ovirt.engine.core.bll.hostdeploy.AddVdsCommand$3.run(AddVdsCommand.java:196) [bll.jar:]
        at org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil$InternalWrapperRunnable.run(ThreadPoolUtil.java:92) [utils.jar:]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [rt.jar:1.8.0_51]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) [rt.jar:1.8.0_51]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_51]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_51]
        at java.lang.Thread.run(Thread.java:745) [rt.jar:1.8.0_51]

2015-08-24 07:30:13,369 ERROR [org.ovirt.engine.core.bll.hostdeploy.InstallVdsInternalCommand] (org.ovirt.thread.pool-7-thread-30) [342f74ca] Host installation failed for host '57a4cb47-0493-49f1-a17e-6c9c12b40f6c', 'alma02.qa.lab.tlv.redhat.com': Command returned failure code 1 during SSH session 'root@alma02.qa.lab.tlv.redhat.com'
2015-08-24 07:30:13,373 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-7-thread-30) [342f74ca] START, SetVdsStatusVDSCommand(HostName = alma02.qa.lab.tlv.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='57a4cb47-0493-49f1-a17e-6c9c12b40f6c', status='InstallFailed', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 71df6a32
2015-08-24 07:30:13,377 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-7-thread-30) [342f74ca] FINISH, SetVdsStatusVDSCommand, log id: 71df6a32
2015-08-24 07:30:13,384 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-7-thread-30) [342f74ca] Correlation ID: 342f74ca, Job ID: dfe150b6-fd51-4903-a8e1-263d1101b4c7, Call Stack: null, Custom Event ID: -1, Message: Host alma02.qa.lab.tlv.redhat.com installation failed. Command returned failure code 1 during SSH session 'root@alma02.qa.lab.tlv.redhat.com'.
2015-08-24 07:30:13,393 INFO  [org.ovirt.engine.core.bll.hostdeploy.InstallVdsInternalCommand] (org.ovirt.thread.pool-7-thread-30) [342f74ca] Lock freed to object 'EngineLock:{exclusiveLocks='[57a4cb47-0493-49f1-a17e-6c9c12b40f6c=<VDS, ACTION_TYPE_FAILED_OBJECT_LOCKED>]', sharedLocks='null'}'

Version-Release number of selected component (if applicable):
Engine running at RHEV3:
Linux version 2.6.32-504.el6.x86_64 (mockbuild@x86-023.build.eng.bos.redhat.com) (gcc version 4.4.7 20120313 (Red Hat 4.4.7-11) (GCC) ) #1 SMP Tue Sep 16 01:56:35 EDT 2014
rhevm-3.6.0-0.12.master.el6.noarch
rhevm-guest-agent-common-1.0.11-1.el6ev.noarch
qemu-guest-agent-0.12.1.2-2.479.el6.x86_64

Host being added:
Linux version 3.10.0-306.0.1.el7.x86_64 (mockbuild@x86-024.build.eng.bos.redhat.com) (gcc version 4.8.3 20140911 (Red Hat 4.8.3-9) (GCC) ) #1 SMP Wed Aug 19 16:06:58 EDT 2015


How reproducible:
100%

Steps to Reproduce:
1.Deploy latest rhevm 3.6 to VM.
2.Freshly install RHEL7.2 to the host.
3.Add host to RHEVM.

Actual results:
Host addition fails with the error: 

Expected results:


Additional info:
logs from engine and sosreport from engine attached.
Comment 1 Nikolai Sednev 2015-08-24 00:53:09 EDT
Created attachment 1066200 [details]
engine_sosreport
Comment 2 Nikolai Sednev 2015-08-24 00:56:21 EDT
Created attachment 1066201 [details]
sosreport_from_host
Comment 3 Alon Bar-Lev 2015-08-24 08:09:09 EDT
At engine.log and audit log:

2015-08-24 07:30:12,919 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) [342f74ca] Correlation ID: 342f74ca, Call Stack: null, Custom Event ID: -1, Message: Failed to install Host alma02.qa.lab.tlv.redhat.com. Failed to execute stage 'Package installation': [u'device-mapper-multipath-0.4.9-81.el7.x86_64 requires kpartx = 0.4.9-81.el7', u'policycoreutils-python-2.2.5-18.el7.x86_64 requires policycoreutils = 2.2.5-18.el7', u'rsyslog-mmjsonparse-7.4.7-10.el7.x86_64 requires rsyslog = 7.4.7-10.el7'].

At host-deploy log:

YumBaseError: [u'device-mapper-multipath-0.4.9-81.el7.x86_64 requires kpartx = 0.4.9-81.el7', u'policycoreutils-python-2.2.5-18.el7.x86_64 requires policycoreutils = 2.2.5-18.el7', u'rsyslog-mmjsonparse-7.4.7-10.el7.x86_64 requires rsyslog = 7.4.7-10.el7']

Has nothing to do with vmconsole.

Not sure why node whiteboard it is not a node.
Comment 4 Nikolai Sednev 2015-08-27 04:30:04 EDT
Now deployment succeeds on these components:
Engine:
ovirt-host-deploy-1.4.0-0.0.4.master.el6ev.noarch
ovirt-engine-extension-aaa-jdbc-0.0.0-4.el6ev.noarch
rhevm-3.6.0-0.12.master.el6.noarch
ovirt-host-deploy-java-1.4.0-0.0.4.master.el6ev.noarch
Red Hat Enterprise Linux Server release 6.7 (Santiago)

Host:
mom-0.5.0-1.el7ev.noarch
sanlock-3.2.4-1.el7.x86_64
libvirt-client-1.2.17-6.el7.x86_64
qemu-kvm-rhev-2.3.0-19.el7.x86_64
ovirt-vmconsole-1.0.0-0.0.master.el7ev.noarch
ovirt-vmconsole-host-1.0.0-0.0.master.el7ev.noarch
vdsm-4.17.3-1.el7ev.noarch
Red Hat Enterprise Linux Server release 7.2 Beta (Maipo)
Comment 5 Yaniv Lavi (Dary) 2015-09-01 07:21:59 EDT
Closing based on comment #4.

Note You need to log in before you can comment on or make changes to this bug.