Bug 1242448
| Summary: | [hosted-engine-setup] Deployment fails due to a sanlock exception creating temporary Posix storage domain on a loopback device | ||
|---|---|---|---|
| Product: | [oVirt] ovirt-hosted-engine-setup | Reporter: | Elad <ebenahar> |
| Component: | General | Assignee: | Simone Tiraboschi <stirabos> |
| Status: | CLOSED CURRENTRELEASE | QA Contact: | Elad <ebenahar> |
| Severity: | urgent | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | --- | CC: | acanan, amureini, bazulay, bugs, ebenahar, ecohen, gklein, jbelka, lsurette, mgoldboi, nsednev, rbalakri, sbonazzo, stirabos, ycui, yeylon |
| Target Milestone: | ovirt-3.6.0-rc | Flags: | rule-engine:
ovirt-3.6.0+
ylavi: planning_ack+ rule-engine: devel_ack+ rule-engine: testing_ack+ |
| Target Release: | 1.3.0 | ||
| Hardware: | x86_64 | ||
| OS: | Unspecified | ||
| Whiteboard: | integration | ||
| Fixed In Version: | Doc Type: | Bug Fix | |
| Doc Text: |
Cause:
The filesystem on the PosixFS loopback device was unlabeled on creation.
Consequence:
SELinux prevents sanlock to use it.
Fix:
Correctly label the PosixFS filesystem
Result:
It works also in SELinux enforcing mode
|
Story Points: | --- |
| Clone Of: | Environment: | ||
| Last Closed: | 2015-11-04 13:36:26 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
| Bug Depends On: | |||
| Bug Blocks: | 1036731, 1153278, 1205663 | ||
Sos report: http://file.tlv.redhat.com/ebenahar/sosreport-green-vdsb.qa.lab.tlv.redhat.com-20150713141553.tar.xz Elad, ban you please check SELinux logs? (In reply to Simone Tiraboschi from comment #2) > Elad, ban you please check SELinux logs? can, sorry :-) Auditd was disabled. Reproduced again with auditd enabled. Here is the sosreport: http://file.tlv.redhat.com/ebenahar/sosreport-green-vdsb.qa.lab.tlv.redhat.com-20150713151711.tar.xz From audit.log:
type=CRED_DISP msg=audit(1436789811.558:357): pid=3672 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:virtd_t:s0-s0:c0.c1023 msg='op=PAM:setcred grantors=pam_env,pam_unix acct="root" exe="/usr/bin/sud
o" hostname=? addr=? terminal=? res=success'
type=AVC msg=audit(1436789811.564:358): avc: denied { read write } for pid=3677 comm="sanlock" name="ids" dev="loop1" ino=16390 scontext=system_u:system_r:sanlock_t:s0-s0:c0.c1023 tcontext=system_u:object_r:unlabeled_t:s0 tclass=file
Does it work in permissive mode? (In reply to Simone Tiraboschi from comment #6) > Does it work in permissive mode? Tested with permissive and it works The HE VM is not started automatically once HE installation is finished.
Checked for its status and got the following exception:
[root@green-vdsb ovirt-hosted-engine-setup]# hosted-engine --vm-status
Traceback (most recent call last):
File "/usr/lib64/python2.7/runpy.py", line 162, in _run_module_as_main
"__main__", fname, loader, pkg_name)
File "/usr/lib64/python2.7/runpy.py", line 72, in _run_code
exec code in run_globals
File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_setup/vm_status.py", line 117, in <module>
if not status_checker.print_status():
File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_setup/vm_status.py", line 60, in print_status
all_host_stats = ha_cli.get_all_host_stats()
File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/client/client.py", line 157, in get_all_host_stats
return self.get_all_stats(self.StatModes.HOST)
File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/client/client.py", line 102, in get_all_stats
stats = broker.get_stats_from_storage(service)
File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", line 232, in get_stats_from_storage
result = self._checked_communicate(request)
File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", line 260, in _checked_communicate
.format(message or response))
ovirt_hosted_engine_ha.lib.exceptions.RequestError: Request failed: <type 'exceptions.OSError'>
So I started the VM manually:
[root@green-vdsb ovirt-hosted-engine-setup]# hosted-engine --vm-start
a7567b69-c29c-46a2-a643-799acf0b1a87
Status = WaitForLaunch
nicModel = rtl8139,pv
statusTime = 4300682480
emulatedMachine = rhel6.5.0
pid = 0
vmName = HostedEngine
devices = [{'index': '2', 'iface': 'ide', 'specParams': {}, 'readonly': 'true', 'deviceId': '1a732367-113d-4e6a-8dcb-9adb45e3e1de', 'address': {'bus': '1', 'controller': '0', 'type': 'drive', 'target': '0', 'unit': '0'}, 'device': 'cdrom', 'shared': 'false', 'path': '', 'type': 'disk'}, {'index': '0', 'iface': 'virtio', 'format': 'raw', 'bootOrder': '1', 'poolID': '00000000-0000-0000-0000-000000000000', 'volumeID': '6a80ef55-6f15-492d-b962-123615bf27cf', 'imageID': 'df02f4f1-e1c7-474b-8075-b839e4bc1c95', 'specParams': {}, 'readonly': 'false', 'domainID': '4a5d3450-655b-452f-8dda-2ef7e051b1a8', 'optional': 'false', 'deviceId': 'df02f4f1-e1c7-474b-8075-b839e4bc1c95', 'address': {'slot': '0x06', 'bus': '0x00', 'domain': '0x0000', 'type': 'pci', 'function': '0x0'}, 'device': 'disk', 'shared': 'exclusive', 'propagateErrors': 'off', 'type': 'disk'}, {'device': 'scsi', 'model': 'virtio-scsi', 'type': 'controller'}, {'nicModel': 'pv', 'macAddr': '00:16:3E:76:D5:D5', 'linkActive': 'true', 'network': 'ovirtmgmt', 'filter': 'vdsm-no-mac-spoofing', 'specParams': {}, 'deviceId': 'a4c22ecc-0e5b-4548-b10a-5ca884d22946', 'address': {'slot': '0x03', 'bus': '0x00', 'domain': '0x0000', 'type': 'pci', 'function': '0x0'}, 'device': 'bridge', 'type': 'interface'}, {'device': 'console', 'specParams': {}, 'type': 'console', 'deviceId': '131f7a43-a609-4795-ba03-9f25f327f6f9', 'alias': 'console0'}]
guestDiskMapping = {}
vmType = kvm
clientIp =
displaySecurePort = -1
memSize = 4096
displayPort = -1
cpuType = Conroe
spiceSecureChannels = smain,sdisplay,sinputs,scursor,splayback,srecord,ssmartcard,susbredir
smp = 2
displayIp = 0
display = vnc
Checked again for the status and got the same exception.
Simone, is this failure to start the VM automatically and the mentioned exception related to the fact I'm working in permissive?
(In reply to Elad from comment #8) > Simone, is this failure to start the VM automatically and the mentioned > exception related to the fact I'm working in permissive? No, it isn't. It looks like a different issue. *** Bug 1238313 has been marked as a duplicate of this bug. *** *** Bug 1247165 has been marked as a duplicate of this bug. *** *** Bug 1225366 has been marked as a duplicate of this bug. *** *** Bug 1247181 has been marked as a duplicate of this bug. *** Hosted-engine deployment over FC is completed successfully. Note that the import of the HE storage domain is blocked due to https://bugzilla.redhat.com/show_bug.cgi?id=1273378 Verified using: ovirt-hosted-engine-setup-1.3.0-1.el7ev.noarch vdsm-4.17.10-5.el7ev.noarch selinux-policy-3.13.1-60.el7.noarch oVirt 3.6.0 has been released on November 4th, 2015 and should fix this issue. If problems still persist, please open a new BZ and reference this one. |
Description of problem: Tried to deploy hosted-engine over FC and it failed due to a Sanlock exception during createStoragePool. Version-Release number of selected component (if applicable): ovirt-3.6.0-3 ovirt-hosted-engine-setup-1.3.0-0.0.master.20150623153111.git68138d4.el7.noarch vdsm-4.17.0-1054.git562e711.el7.noarch sanlock-3.2.2-2.el7.x86_64 How reproducible: Always Steps to Reproduce: 1. Deploy hosted-engine using FC 2. 3. Actual results: The deployment fails in createStoragePool due to a sanlock exception: Setup.log 2015-07-13 14:00:30 DEBUG otopi.context context._executeMethod:155 method exception Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/otopi/context.py", line 145, in _executeMethod method['method']() File "/usr/share/ovirt-hosted-engine-setup/scripts/../plugins/ovirt-hosted-engine-setup/storage/storage.py", line 1212, in _misc self._createStoragePool() File "/usr/share/ovirt-hosted-engine-setup/scripts/../plugins/ovirt-hosted-engine-setup/storage/storage.py", line 783, in _createStoragePool raise RuntimeError(status['status']['message']) RuntimeError: Cannot acquire host id: ('e45c292e-3167-4efc-ac3a-ce2e216f6590', SanlockException(19, 'Sanlock lockspace add failure', 'No such device')) 2015-07-13 14:00:30 ERROR otopi.context context._executeMethod:164 Failed to execute stage 'Misc configuration': Cannot acquire host id: ('e45c292e-3167-4efc-ac3a-ce2e216f6590', SanlockException(19, 'Sanlock locks pace add failure', 'No such device')) vdsm.log: Thread-40::ERROR::2015-07-13 14:00:30,338::task::866::Storage.TaskManager.Task::(_setError) Task=`297d0697-6c4e-4967-a65e-c17ac2891810`::Unexpected error Traceback (most recent call last): File "/usr/share/vdsm/storage/task.py", line 873, in _run return fn(*args, **kargs) File "/usr/share/vdsm/logUtils.py", line 49, in wrapper res = f(*args, **kwargs) File "/usr/share/vdsm/storage/hsm.py", line 1003, in createStoragePool leaseParams) File "/usr/share/vdsm/storage/sp.py", line 580, in create self._acquireTemporaryClusterLock(msdUUID, leaseParams) File "/usr/share/vdsm/storage/sp.py", line 512, in _acquireTemporaryClusterLock msd.acquireHostId(self.id) File "/usr/share/vdsm/storage/sd.py", line 477, in acquireHostId self._clusterLock.acquireHostId(hostId, async) File "/usr/share/vdsm/storage/clusterlock.py", line 237, in acquireHostId raise se.AcquireHostIdFailure(self._sdUUID, e) AcquireHostIdFailure: Cannot acquire host id: ('e45c292e-3167-4efc-ac3a-ce2e216f6590', SanlockException(19, 'Sanlock lockspace add failure', 'No such device')) Sanlock.log: 2015-07-13 14:00:29+0300 899 [672]: s1 lockspace e45c292e-3167-4efc-ac3a-ce2e216f6590:250:/rhev/data-center/mnt/_var_lib_ovirt-hosted-engine-setup_tmp9HMYnA/e45c292e-3167-4efc-ac3a-ce2e216f6590/dom_md/ids:0 2015-07-13 14:00:29+0300 899 [3842]: open error -13 /rhev/data-center/mnt/_var_lib_ovirt-hosted-engine-setup_tmp9HMYnA/e45c292e-3167-4efc-ac3a-ce2e216f6590/dom_md/ids 2015-07-13 14:00:29+0300 899 [3842]: s1 open_disk /rhev/data-center/mnt/_var_lib_ovirt-hosted-engine-setup_tmp9HMYnA/e45c292e-3167-4efc-ac3a-ce2e216f6590/dom_md/ids error -13 2015-07-13 14:00:30+0300 900 [672]: s1 add_lockspace fail result -19 Expected results: Deployment should succeed over FC Additional info: sosreport