Bug 1124762

Summary: Unexpected error while setting up hosted-engine on EL7
Product: Red Hat Enterprise Virtualization Manager Reporter: Sandro Bonazzola <sbonazzo>
Component: vdsmAssignee: Allon Mureinik <amureini>
Status: CLOSED CURRENTRELEASE QA Contact: Artyom <alukiano>
Severity: urgent Docs Contact:
Priority: unspecified    
Version: 3.5.0CC: alukiano, amureini, bazulay, dfediuck, didi, ecohen, gklein, iheim, lpeer, lveyde, msivak, pstehlik, sbonazzo, scohen, sherold, smizrahi, stirabos, yeylon
Target Milestone: ---Keywords: TestOnly
Target Release: 3.5.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard: storage
Fixed In Version: vt4.1 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2015-02-16 13:39:40 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: Storage RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Bug Depends On:    
Bug Blocks: 1147536, 1156162, 1164308, 1164311    
Attachments:
Description Flags
logs
none
vdsm.log none

Description Sandro Bonazzola 2014-07-30 09:18:29 UTC
Created attachment 922471 [details]
logs

Trying to test hosted-engine on EL7, I got the following Traceback in VDSM:


 Thread-52::ERROR::2014-07-30 11:09:28,688::task::866::Storage.TaskManager.Task::(_setError) Task=`a041a63b-ef51-4352-98de-c5169bf8dcf1`::Unexpected error
 Traceback (most recent call last):
  File "/usr/share/vdsm/storage/task.py", line 873, in _run
    return fn(*args, **kargs)
  File "/usr/share/vdsm/logUtils.py", line 45, in wrapper
    res = f(*args, **kwargs)
  File "/usr/share/vdsm/storage/hsm.py", line 3158, in getVolumePath
    volUUID=volUUID).getVolumePath()
  File "/usr/share/vdsm/storage/sd.py", line 400, in produceVolume
    volUUID)
  File "/usr/share/vdsm/storage/fileVolume.py", line 58, in __init__
    volume.Volume.__init__(self, repoPath, sdUUID, imgUUID, volUUID)
  File "/usr/share/vdsm/storage/volume.py", line 144, in __init__
    self.validate()
  File "/usr/share/vdsm/storage/volume.py", line 156, in validate
    self.validateImagePath()
  File "/usr/share/vdsm/storage/fileVolume.py", line 487, in validateImagePath
    if not self.oop.os.path.isdir(imageDir):
  File "/usr/share/vdsm/storage/outOfProcess.py", line 251, in isdir
    res = self._iop.stat(path)
  File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 367, in stat
    resdict = self._sendCommand("stat", {"path": path}, self.timeout)
  File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 344, in _sendCommand
    raise OSError(errcode, errstr)
 OSError: [Errno 2] No such file or directory

Storage is NFSv3 on a different host.

Comment 3 Allon Mureinik 2014-09-02 07:29:48 UTC
Saggie, do we have an ioprocess build available on EL7?

Comment 5 Barak 2014-09-21 15:20:33 UTC
There is an available package for ioprocess for EL7,
This should be handled by storage group , moving back to storage.

BTW the initial error is "No such file or directory" ...

Comment 6 Yoav Kleinberger 2014-09-28 11:32:53 UTC
Is this an engine running on a RHEL7 VM, or an engine running on any VM operating system, but the *host* is running RHEL7?

I'm pretty sure you meant the host, but I want to be sure.

Comment 7 Artyom 2014-09-28 11:45:07 UTC
Reporter is Sandro so the best way it ask him, but how I understand host is EL7.
Also hard to understand when it happen, on what stage, on deployment of hosted engine or when hosted-engine deployment already finished, Sandro can you give a little more details about the flow, when exception appear, what actions preceded.

Comment 8 Sandro Bonazzola 2014-09-29 05:59:09 UTC
This is ovirt-hosted-engine-setup running on EL7 for creating a VM where it would have been installed EL 6.5 + RHEV-M 3.5 (but it failed before creating the VM)

It failed at Volume creation stage, after having created the storage domain:

2014-07-30 11:09:28 DEBUG otopi.context context._executeMethod:152 method exception
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/otopi/context.py", line 142, in _executeMethod
    method['method']()
  File "/usr/share/ovirt-hosted-engine-setup/scripts/../plugins/ovirt-hosted-engine-setup/sanlock/lockspace.py", line 163, in _misc
    lockspace + '.metadata': md_size,
  File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/storage_backends.py", line 336, in create
    service_size=size)
  File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/storage_backends.py", line 268, in create_volume
    raise RuntimeError(response["status"]["message"])
RuntimeError: [Errno 2] No such file or directory

There are full logs of the setup execution and corresponding vdsm logs in the attached archive.

Comment 9 Sandro Bonazzola 2014-10-02 12:32:21 UTC
I'm not able to reproduce the issue using upstream rpms,maybe we should move this to QE as test-only so it can be checked on next downstream build?

Comment 10 Allon Mureinik 2014-10-02 12:59:26 UTC
(In reply to Sandro Bonazzola from comment #9)
> I'm not able to reproduce the issue using upstream rpms,maybe we should move
> this to QE as test-only so it can be checked on next downstream build?
Agreed.

Comment 11 Artyom 2014-10-05 14:35:51 UTC
I finish deployment for vt4 and not encounter this kind of exception, but all vdsm logs full of:
Thread-20::ERROR::2014-10-05 17:23:10,961::sampling::475::vm.Vm::(collect) vmId=`f4675766-a472-4c49-bde7-ad68db30e77c`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x2adf1b0>
Traceback (most recent call last):
  File "/usr/share/vdsm/virt/sampling.py", line 471, in collect
    statsFunction()
  File "/usr/share/vdsm/virt/sampling.py", line 346, in __call__
    retValue = self._function(*args, **kwargs)
  File "/usr/share/vdsm/virt/vm.py", line 313, in _sampleCpu
    cpuStats = self._vm._dom.getCPUStats(True, 0)
  File "/usr/share/vdsm/virt/vm.py", line 670, in f
    ret = attr(*args, **kwargs)
  File "/usr/lib64/python2.7/site-packages/vdsm/libvirtconnection.py", line 111, in wrapper
    ret = f(*args, **kwargs)
  File "/usr/lib64/python2.7/site-packages/libvirt.py", line 2016, in getCPUStats
    if ret is None: raise libvirtError ('virDomainGetCPUStats() failed', dom=self)
libvirtError: Requested operation is not valid: cgroup CPUACCT controller is not mounted
Thread-20::DEBUG::2014-10-05 17:23:10,964::libvirtconnection::143::root::(wrapper) Unknown libvirterror: ecode: 55 edom: 10 level: 2 message: Requested operation is not valid: cgroup CPU controller is not mounted

So I can reassign this bug or just open one new, what you prefer?

Comment 12 Artyom 2014-10-05 14:37:20 UTC
Created attachment 944045 [details]
vdsm.log

Comment 13 Allon Mureinik 2014-10-05 15:29:37 UTC
(In reply to Artyom from comment #11)
> So I can reassign this bug or just open one new, what you prefer?
The two issues seem unrelated - please open a new BZ.

Comment 14 Artyom 2014-10-06 07:54:21 UTC
Verified on vdsm-4.16.5-2.el7.x86_64 and ovirt-hosted-engine-setup-1.2.0-1.el7.noarch