Bug 1124762 - Unexpected error while setting up hosted-engine on EL7
Summary: Unexpected error while setting up hosted-engine on EL7
Alias: None
Product: Red Hat Enterprise Virtualization Manager
Classification: Red Hat
Component: vdsm
Version: 3.5.0
Hardware: Unspecified
OS: Unspecified
Target Milestone: ---
: 3.5.0
Assignee: Allon Mureinik
QA Contact: Artyom
Whiteboard: storage
Depends On:
Blocks: rhev35betablocker rhev3.5beta3 rhev35rcblocker rhev35gablocker
TreeView+ depends on / blocked
Reported: 2014-07-30 09:18 UTC by Sandro Bonazzola
Modified: 2016-02-10 17:15 UTC (History)
18 users (show)

Fixed In Version: vt4.1
Doc Type: Bug Fix
Doc Text:
Clone Of:
Last Closed: 2015-02-16 13:39:40 UTC
oVirt Team: Storage
Target Upstream Version:

Attachments (Terms of Use)
logs (104.25 KB, application/x-compressed-tar)
2014-07-30 09:18 UTC, Sandro Bonazzola
no flags Details
vdsm.log (16.46 MB, text/plain)
2014-10-05 14:37 UTC, Artyom
no flags Details

Description Sandro Bonazzola 2014-07-30 09:18:29 UTC
Created attachment 922471 [details]

Trying to test hosted-engine on EL7, I got the following Traceback in VDSM:

 Thread-52::ERROR::2014-07-30 11:09:28,688::task::866::Storage.TaskManager.Task::(_setError) Task=`a041a63b-ef51-4352-98de-c5169bf8dcf1`::Unexpected error
 Traceback (most recent call last):
  File "/usr/share/vdsm/storage/task.py", line 873, in _run
    return fn(*args, **kargs)
  File "/usr/share/vdsm/logUtils.py", line 45, in wrapper
    res = f(*args, **kwargs)
  File "/usr/share/vdsm/storage/hsm.py", line 3158, in getVolumePath
  File "/usr/share/vdsm/storage/sd.py", line 400, in produceVolume
  File "/usr/share/vdsm/storage/fileVolume.py", line 58, in __init__
    volume.Volume.__init__(self, repoPath, sdUUID, imgUUID, volUUID)
  File "/usr/share/vdsm/storage/volume.py", line 144, in __init__
  File "/usr/share/vdsm/storage/volume.py", line 156, in validate
  File "/usr/share/vdsm/storage/fileVolume.py", line 487, in validateImagePath
    if not self.oop.os.path.isdir(imageDir):
  File "/usr/share/vdsm/storage/outOfProcess.py", line 251, in isdir
    res = self._iop.stat(path)
  File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 367, in stat
    resdict = self._sendCommand("stat", {"path": path}, self.timeout)
  File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 344, in _sendCommand
    raise OSError(errcode, errstr)
 OSError: [Errno 2] No such file or directory

Storage is NFSv3 on a different host.

Comment 3 Allon Mureinik 2014-09-02 07:29:48 UTC
Saggie, do we have an ioprocess build available on EL7?

Comment 5 Barak 2014-09-21 15:20:33 UTC
There is an available package for ioprocess for EL7,
This should be handled by storage group , moving back to storage.

BTW the initial error is "No such file or directory" ...

Comment 6 Yoav Kleinberger 2014-09-28 11:32:53 UTC
Is this an engine running on a RHEL7 VM, or an engine running on any VM operating system, but the *host* is running RHEL7?

I'm pretty sure you meant the host, but I want to be sure.

Comment 7 Artyom 2014-09-28 11:45:07 UTC
Reporter is Sandro so the best way it ask him, but how I understand host is EL7.
Also hard to understand when it happen, on what stage, on deployment of hosted engine or when hosted-engine deployment already finished, Sandro can you give a little more details about the flow, when exception appear, what actions preceded.

Comment 8 Sandro Bonazzola 2014-09-29 05:59:09 UTC
This is ovirt-hosted-engine-setup running on EL7 for creating a VM where it would have been installed EL 6.5 + RHEV-M 3.5 (but it failed before creating the VM)

It failed at Volume creation stage, after having created the storage domain:

2014-07-30 11:09:28 DEBUG otopi.context context._executeMethod:152 method exception
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/otopi/context.py", line 142, in _executeMethod
  File "/usr/share/ovirt-hosted-engine-setup/scripts/../plugins/ovirt-hosted-engine-setup/sanlock/lockspace.py", line 163, in _misc
    lockspace + '.metadata': md_size,
  File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/storage_backends.py", line 336, in create
  File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/storage_backends.py", line 268, in create_volume
    raise RuntimeError(response["status"]["message"])
RuntimeError: [Errno 2] No such file or directory

There are full logs of the setup execution and corresponding vdsm logs in the attached archive.

Comment 9 Sandro Bonazzola 2014-10-02 12:32:21 UTC
I'm not able to reproduce the issue using upstream rpms,maybe we should move this to QE as test-only so it can be checked on next downstream build?

Comment 10 Allon Mureinik 2014-10-02 12:59:26 UTC
(In reply to Sandro Bonazzola from comment #9)
> I'm not able to reproduce the issue using upstream rpms,maybe we should move
> this to QE as test-only so it can be checked on next downstream build?

Comment 11 Artyom 2014-10-05 14:35:51 UTC
I finish deployment for vt4 and not encounter this kind of exception, but all vdsm logs full of:
Thread-20::ERROR::2014-10-05 17:23:10,961::sampling::475::vm.Vm::(collect) vmId=`f4675766-a472-4c49-bde7-ad68db30e77c`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x2adf1b0>
Traceback (most recent call last):
  File "/usr/share/vdsm/virt/sampling.py", line 471, in collect
  File "/usr/share/vdsm/virt/sampling.py", line 346, in __call__
    retValue = self._function(*args, **kwargs)
  File "/usr/share/vdsm/virt/vm.py", line 313, in _sampleCpu
    cpuStats = self._vm._dom.getCPUStats(True, 0)
  File "/usr/share/vdsm/virt/vm.py", line 670, in f
    ret = attr(*args, **kwargs)
  File "/usr/lib64/python2.7/site-packages/vdsm/libvirtconnection.py", line 111, in wrapper
    ret = f(*args, **kwargs)
  File "/usr/lib64/python2.7/site-packages/libvirt.py", line 2016, in getCPUStats
    if ret is None: raise libvirtError ('virDomainGetCPUStats() failed', dom=self)
libvirtError: Requested operation is not valid: cgroup CPUACCT controller is not mounted
Thread-20::DEBUG::2014-10-05 17:23:10,964::libvirtconnection::143::root::(wrapper) Unknown libvirterror: ecode: 55 edom: 10 level: 2 message: Requested operation is not valid: cgroup CPU controller is not mounted

So I can reassign this bug or just open one new, what you prefer?

Comment 12 Artyom 2014-10-05 14:37:20 UTC
Created attachment 944045 [details]

Comment 13 Allon Mureinik 2014-10-05 15:29:37 UTC
(In reply to Artyom from comment #11)
> So I can reassign this bug or just open one new, what you prefer?
The two issues seem unrelated - please open a new BZ.

Comment 14 Artyom 2014-10-06 07:54:21 UTC
Verified on vdsm-4.16.5-2.el7.x86_64 and ovirt-hosted-engine-setup-1.2.0-1.el7.noarch

Note You need to log in before you can comment on or make changes to this bug.