Created attachment 1094126 [details] Engine log (copied about 15 minutes after I saw the error) Description of problem: UI display this error when putting host to maintenance: Error while executing action: A Request to the Server failed: java.lang.reflect.InvocationTargetException Host do move to maintenance. Version-Release number of selected component (if applicable): oVirt Engine Version: 3.6.0-0.0.master.20150915165908.git51dfb6e.fc22 Browser: Chrome Version 46.0.2490.86 (64-bit) OS: Fedora 22 (kernel 4.2.5-201.fc22.x86_64) How reproducible: Sometimes. Steps to Reproduce: This happened several times in the past. This is what I did when it happened today: 1. Try to migrate vm 2. Migration fails (see bug 1282054) 3. Shutdown vm 4. Remove vm 5. Put non-spm host to maintenance Error displayed, host move to maintenance 6. Put spm host to maintenance Error displayed, host move to maintenance Expected results: Should not display errors when everything seems to work.
Tested again with newer engine: oVirt Engine Version: 3.6.1-0.0.master.20151113175558.git6a36a6d.fc22 Did same flow, error not displayed. Maybe this was fixed, or maybe something else is needed to reproduce it.
Host maintenance is an infra flow. Setting that whiteboard for initial investigation.
Moti - please take a look. If reproducible - fix. If not, close. Thanks
Tried couple of times - didn't experience the described error message. Usually additional information for UI related errors could be found on server.log.
Reproduced again with: oVirt Engine Version: 3.6.1-0.0.master.20151113175558.git6a36a6d.fc22 In a setup with two hosts, select both hosts, click "maintenance", error pops up, or maybe it pops up after clicking ok.
Created attachment 1100002 [details] Server log
Created attachment 1100003 [details] Engine log
The error occurred between 7pm and 9pm.
The server.log contains a violation key exception in host-devices table: Caused by: org.postgresql.util.PSQLException: ERROR: insert or update on table "host_device" violates foreign key constraint "fk_host_device_parent_name" Detail: Key (host_id, parent_device_name)=(89887385-48b3-4648-b397-f2078519a61e, scsi_host3) is not present in table "host_device". The foreign key is defined as: SELECT fn_db_create_constraint('host_device', 'fk_host_device_parent_name', 'FOREIGN KEY (host_id, parent_device_name) REFERENCES host_device(host_id, device_name) DEFERRABLE INITIALLY IMMEDIATE'); Martin, could you advise of the above ?
It seems the VDSM is reporting inconsistent data for host devices. From the return value of HostDevListByCapsVDSCommand we can see that no 'scsi_host3' device was reported although it is being referenced as parent of 'scsi_target3_0_0' device. Adding Martin Polednik to investigate. @Nir: please try to provide relevant VDSM logs if still available.
As Martin requested, we'll need to see the VDSM logs. There shouldn't be any kind of device filtering (except for caps) - VDSM returns devices that libvirt reports. Also, make sure that there is no hook (after_hostdev_list_by_caps) interfering with the call. It would also be helpful to see 'virsh -r nodedev-list'.
(In reply to Martin Betak from comment #10) > @Nir: please try to provide relevant VDSM logs if still available. I don't have the vdsm logs, they were deleted.
(In reply to Martin Polednik from comment #11) > It would also be helpful to see 'virsh -r nodedev-list'. I can provide it but not with the vdsm version used in the bug, and I'm not sure even about rhel/libvirt version, I upgraded this host to 7.2.
I don't see how we can resolve this without the information requested in the latest comments. Closing. If it reproduces, please re-open with all relevant logs, and in addition the output of virsh -r nodedev-list, as Martin requested.