Bug 1268223

Summary: VDSM doesn't start due to failing dependency
Product: [oVirt] vdsm Reporter: Christopher Pereira <kripper>
Component: ServicesAssignee: Yaniv Bronhaim <ybronhei>
Status: CLOSED NOTABUG QA Contact: Aharon Canan <acanan>
Severity: urgent Docs Contact:
Priority: unspecified    
Version: 4.17.0CC: amureini, bugs, kripper, msivak, sbonazzo
Target Milestone: ovirt-3.6.0-rc3Flags: amureini: ovirt-3.6.0?
rule-engine: planning_ack?
rule-engine: devel_ack?
rule-engine: testing_ack?
Target Release: 4.17.8   
Hardware: x86_64   
OS: Linux   
Whiteboard: storage
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2015-10-07 16:08:03 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: Storage RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
vdsm.log
none
ovirt-hosted-engine-setup-20151002051145-j0j7tn.log
none
sosreport none

Description Christopher Pereira 2015-10-02 08:20:10 UTC
Created attachment 1079385 [details]
vdsm.log

[ INFO  ] Stage: Transaction setup
[ INFO  ] Stage: Misc configuration
[ INFO  ] Stage: Package installation
[ INFO  ] Stage: Misc configuration
[ INFO  ] Configuring libvirt
[ INFO  ] Configuring VDSM
[ INFO  ] Starting vdsmd
[ ERROR ] Failed to execute stage 'Misc configuration': Failed to start service 'vdsmd'
[ INFO  ] Stage: Clean up
[ INFO  ] Generating answer file '/var/lib/ovirt-hosted-engine-setup/answers/answers-20151002051315.conf'
[ INFO  ] Stage: Pre-termination
[ INFO  ] Stage: Termination

---

Attached relevant logs

---

# rpm -qa |grep ovirt
ovirt-setup-lib-1.0.0-0.0.master.20150812132738.git6a54bc0.el7.centos.noarch
ovirt-engine-appliance-20150802.0-1.el7.centos.noarch
ovirt-vmconsole-host-1.0.0-0.0.master.20150821105434.gite14b2f0.el7.noarch
ovirt-hosted-engine-setup-1.3.0-0.0.master.20150914144133.gitc63047c.el7.centos.noarch
libgovirt-0.3.1-3.el7.x86_64
ovirt-hosted-engine-ha-1.3.0-0.0.master.20150909150556.20150909150548.git9a2bd43.el7.noarch
ovirt-vmconsole-1.0.0-0.0.master.20150821105434.gite14b2f0.el7.noarch
ovirt-release36-001-0.5.beta.noarch
ovirt-host-deploy-1.4.0-0.0.master.20150806005708.git670e9c8.el7.noarch
ovirt-engine-sdk-python-3.6.0.2-1.el7.centos.noarch

Comment 1 Christopher Pereira 2015-10-02 08:21:44 UTC
Created attachment 1079386 [details]
ovirt-hosted-engine-setup-20151002051145-j0j7tn.log

Comment 2 Sandro Bonazzola 2015-10-02 08:45:39 UTC
Can you please attach full sos report?

Comment 3 Christopher Pereira 2015-10-02 14:33:59 UTC
Created attachment 1079462 [details]
sosreport

Comment 4 Christopher Pereira 2015-10-04 04:06:39 UTC
VDSMd receives SIGTERM 15 (kill) during hosted-engine setup.
CGROUP related?

Comment 5 Sandro Bonazzola 2015-10-07 15:27:14 UTC
2015-10-02 05:13:15 DEBUG otopi.plugins.otopi.services.systemd plugin.execute:941 execute-output: ('/bin/systemctl', 'start', 'vdsmd.service') stderr:
A dependency job for vdsmd.service failed. See 'journalctl -xn' for details.

SuperVDSM started correctly:
MainThread::DEBUG::2015-10-02 05:13:15,470::supervdsmServer::520::SuperVdsm.Server::(main) Making sure I'm root - SuperVdsm

But VDSM didn't start.
Looking at /var/log/messages and at systemd status looks like the following services failed to start:
mnt-datacenter.mount  loaded failed failed IMATRONIX DataCenter Mount
ix-datacenter.service loaded failed failed IMATRONIX DataCenter

mom process stopped earlier and never restart:
2015-10-02 03:07:04,596 - mom - INFO - MOM ending

In /var/log/messages I see:
Oct  2 04:01:13 h8 mount: Mount failed. Please check the log file for more details.
Oct  2 04:01:13 h8 systemd: Failed at step EXEC spawning /mnt/datacenter/bin/init-datacenter: Transport endpoint is not connected
Oct  2 04:01:13 h8 systemd: mnt-datacenter.mount mount process exited, code=exited status=1
Oct  2 04:01:13 h8 systemd: Unit mnt-datacenter.mount entered failed state.
Oct  2 04:01:13 h8 systemd: ix-datacenter.service: main process exited, code=exited, status=203/EXEC
Oct  2 04:01:13 h8 systemd: Failed to start IMATRONIX DataCenter.
Oct  2 04:01:13 h8 systemd: Dependency failed for Virtual Desktop Server Manager.
Oct  2 04:01:13 h8 systemd: Dependency failed for MOM instance configured for VDSM purposes.
Oct  2 04:01:13 h8 systemd: 
Oct  2 04:01:13 h8 systemd: 


I think that your disk mount failure is causing the vdsm dependency failure here.
Can you fix the monut issue and retry?
I'm moving this to VDSM component, allowing them to review the issue as well.

Comment 6 Christopher Pereira 2015-10-07 16:08:03 UTC
Could be, but the strange thing is that starting VDSMD via "systemctl start vdsmd" works fine, even when the other service is throwing an error because of the mount.

Closing for now as "NOTABUG", until I can confirm on another host (this one is in production now).

Thanks for your time, Sandro.

Comment 7 Sandro Bonazzola 2015-10-09 10:09:14 UTC
You're welcome Christopher.