Bug 1281363

How high is the reproducebility?

For me it happen always, I also want to specify that host pass upgrade from
Red Hat Enterprise Virtualization Hypervisor release 6.7 (20151028.0.el6ev) to
Red Hat Enterprise Virtualization Hypervisor release 6.7 (20151029.0.el6ev), so maybe it reason why it failed, because I found that we encountered the same problem in past
http://comments.gmane.org/gmane.comp.emulators.ovirt.user/27482

Please attach a complete sosreport and the output of "ps eZ"

vdsm-4.16.28-1.el6ev.x86_64 should have been 4.16.29. Please check the RHEV-H content.

moving downstream being this downstream only bug

Created attachment 1093267 [details]
sos report

Just take in account we now have problem with VNX storage, and it will be many errors about it in logs.

Created attachment 1093268 [details]
output of ps eZ

Sorry, ps -eZ please

See in attachments

Created attachment 1093276 [details]
output of ps -eZ

libvirtd was started in the wrong context:

system_u:system_r:ovirt_t:s0    14682 ?        00:00:04 libvirtd

It should be in system_u:system_r:virtd_t:s0-s0:c0.c1023 or alike.

I actually wonder if this is the root cause, because that message appears somewhere in the middle, at the end there are several other errors:

Nov 12 10:39:31 localhost vdsm vds ERROR failed to retrieve Hosted Engine HA info#012Traceback (most recent call last):#012  File "/usr/share/vdsm/API.py", line 1707, in _getHaInfo#012  File "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/client/client.py", line 100, in get_all_stats#012  File "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/client/client.py", line 177, in _configure_broker_conn#012  File "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", line 176, in set_storage_domain#012RequestError: Failed to set storage domain FilesystemBackend, options {'dom_type': 'nfs3', 'sd_uuid': '09035925-5377-4024-919e-880eee080fa0'}: Request failed: <class 'ovirt_hosted_engine_ha.lib.storage_backends.BackendFailureException'>
Nov 12 10:39:31 localhost ovirt-ha-broker ovirt_hosted_engine_ha.broker.listener.ConnectionHandler ERROR Error handling request, data: 'set-storage-domain FilesystemBackend dom_type=nfs3 sd_uuid=09035925-5377-4024-919e-880eee080fa0'#012Traceback (most recent call last):#012  File "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/broker/listener.py", line 166, in handle#012  File "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/broker/listener.py", line 299, in _dispatch#012  File "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/broker/storage_broker.py", line 65, in set_storage_domain#012  File "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/lib/storage_backends.py", line 450, in connect#012  File "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/lib/storage_backends.py", line 107, in get_domain_path#012BackendFailureException: path to storage domain 09035925-5377-4024-919e-880eee080fa0 not found in /rhev/data-center/mnt
Nov 12 10:39:31 localhost vdsm vds ERROR failed to retrieve Hosted Engine HA info#012Traceback (most recent call last):#012  File "/usr/share/vdsm/API.py", line 1707, in _getHaInfo#012  File "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/client/client.py", line 100, in get_all_stats#012  File "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/client/client.py", line 177, in _configure_broker_conn#012  File "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", line 176, in set_storage_domain#012RequestError: Failed to set storage domain FilesystemBackend, options {'dom_type': 'nfs3', 'sd_uuid': '09035925-5377-4024-919e-880eee080fa0'}: Request failed: <class 'ovirt_hosted_engine_ha.lib.storage_backends.BackendFailureException'>
Nov 12 10:39:41 localhost ovirt-ha-broker ovirt_hosted_engine_ha.broker.listener.ConnectionHandler ERROR Error handling request, data: 'set-storage-domain FilesystemBackend dom_type=nfs3 sd_uuid=09035925-5377-4024-919e-880eee080fa0'#012Traceback (most recent call last):#012  File "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/broker/listener.py", line 166, in handle#012  File "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/broker/listener.py", line 299, in _dispatch#012  File "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/broker/storage_broker.py", line 65, in set_storage_domain#012  File "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/lib/storage_backends.py", line 450, in connect#012  File "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/lib/storage_backends.py", line 107, in get_domain_path#012BackendFailureException: path to storage domain 09035925-5377-4024-919e-880eee080fa0 not found in /rhev/data-center/mnt
Nov 12 10:39:41 localhost vdsm vds ERROR failed to retrieve Hosted Engine HA info#012Traceback (most recent call last):#012  File "/usr/share/vdsm/API.py", line 1707, in _getHaInfo#012  File "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/client/client.py", line 100, in get_all_stats#012  File "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/client/client.py", line 177, in _configure_broker_conn#012  File "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", line 176, in set_storage_domain#012RequestError: Failed to set storage domain FilesystemBackend, options {'dom_type': 'nfs3', 'sd_uuid': '09035925-5377-4024-919e-880eee080fa0'}: Request failed: <class 'ovirt_hosted_engine_ha.lib.storage_backends.BackendFailureException'>
Nov 12 10:39:51 localhost ovirt-ha-broker ovirt_hosted_engine_ha.broker.listener.ConnectionHandler ERROR Error handling request, data: 'set-storage-domain FilesystemBackend dom_type=nfs3 sd_uuid=09035925-5377-4024-919e-880eee080fa0'#012Traceback (most recent call last):#012  File "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/broker/listener.py", line 166, in handle#012  File "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/broker/listener.py", line 299, in _dispatch#012  File "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/broker/storage_broker.py", line 65, in set_storage_domain#012  File "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/lib/storage_backends.py", line 450, in connect#012  File "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/lib/storage_backends.py", line 107, in get_domain_path#012BackendFailureException: path to storage domain 09035925-5377-4024-919e-880eee080fa0 not found in /rhev/data-center/mnt
Nov 12 10:39:51 localhost vdsm vds ERROR failed to retrieve Hosted Engine HA info#012Traceback (most recent call last):#012  File "/usr/share/vdsm/API.py", line 1707, in _getHaInfo#012  File "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/client/client.py", line 100, in get_all_stats#012  File "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/client/client.py", line 177, in _configure_broker_conn#012  File "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", line 176, in set_storage_domain#012RequestError: Failed to set storage domain FilesystemBackend, options {'dom_type': 'nfs3', 'sd_uuid': '09035925-5377-4024-919e-880eee080fa0'}: Request failed: <class 'ovirt_hosted_engine_ha.lib.storage_backends.BackendFailureException'>

Here it seams that the storage wasn't available:

Nov 12 10:39:41 localhost ovirt-ha-broker ovirt_hosted_engine_ha.broker.listener.ConnectionHandler ERROR Error handling request, data: 'set-storage-domain FilesystemBackend dom_type=nfs3 sd_uuid=09035925-5377-4024-919e-880eee080fa0'#012Traceback (most recent call last):#012  File "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/broker/listener.py", line 166, in handle#012  File "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/broker/listener.py", line 299, in _dispatch#012  File "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/broker/storage_broker.py", line 65, in set_storage_domain#012  File "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/lib/storage_backends.py", line 450, in connect#012  File "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/lib/storage_backends.py", line 107, in get_domain_path#012BackendFailureException: path to storage domain 09035925-5377-4024-919e-880eee080fa0 not found in /rhev/data-center/mnt

Thanks.

Artyom, can you please retry to reproduce this bug on other hardware and ensure that the storage is reachable?

I reproduced this bug after upgrade to hypervisor6-6.7-20151029.0 or hypervisor6-6.7-20151112.0, but not 100% reproduce. I will give more info soon because testing is in process, I need to try more scenarios and different hardware.

(In reply to Fabian Deutsch from comment #16)
> Thanks.
> 
> Artyom, can you please retry to reproduce this bug on other hardware and
> ensure that the storage is reachable?

Let me instead on this needinfo.
Yes, storage is reachable, Host and Storage is active.

Created attachment 1093584 [details]
sosreport_iSCSStorage_rhevh67_20151112.0

Created attachment 1093585 [details]
freshlog_varlog_rhevh67_20151112.0

Created attachment 1093586 [details]
screenshot_VM_Failed_Start

Created attachment 1093588 [details]
engine.log

Created attachment 1093589 [details]
ps -eZ output

comment 20 - 24 for fresh logs info, for further investigation.

The cause is probably (as noted above) that libvirtd is running in the wrong context:

[root@dhcp-test-9-205 ~]# ps -eZ | grep libvi
system_u:system_r:ovirt_t:s0    14740 ?        00:00:00 libvirtd

The correct context would be like:

[root@dhcp-test-9-205 ~]# ps -eZ | grep libvirtd
system_u:system_r:virtd_t:s0-s0:c0.c1023 14708 ? 00:00:00 libvirtd

I modified the x.x.x.205 host and the VMs now come up again.

The attached patch fixes this in the code.

This bug only happened on RHEV-H el6, no need clone to zstream. 
so I verified this bug on rhevh 6.7 for 3.5.6

# cat /etc/redhat-release 
Red Hat Enterprise Virtualization Hypervisor release 6.7 (20151117.0.el6ev)

# rpm -qa ovirt-node
ovirt-node-3.2.3-28.el6.noarch

Test servers:
Local Storage - 2 times, PASS
iSCSI Storage - 2 times, PASS


Test Steps:
1. Install RHEV-H 6.7-20151015.1 the last released version
2. Register to RHEVM 3.5.6.2-0.1.el6ev successful
3. Connect storage
4. Create VM, VM can start successful.
5. Shutdown VM
6. Make Host to maintenance
7. Upgrade RHEVH 6.7 20151117.0 via RHEVM
8. Upgrade successful.
9. Host and Storage is active yet.
10. Start VM which created in step 4 - PASS.
11. Create new VM, and start VM - PASS.

I will change the bug status once the bug is ON_QA.

This bug only happened on RHEV-H 6.7 el6, so we can not verify it on RHEV-H 7.2 for 3.6.0 build.

This issue was verified on RHEV-H 6.7 for 3.5.6 build yet.

Suggest to close this bug. Thanks

Closing this bug according to comment 32.

Thanks Ying.

This bug is valid bug when bug is reported, but just for 3.5.z, So updated the status to current release instead of notabug.