Bug 1282067 - Error when putting host to maintenance: Error while executing action: A Request to the Server failed: java.lang.reflect.InvocationTargetException
Error when putting host to maintenance: Error while executing action: A Reque...
Status: CLOSED INSUFFICIENT_DATA
Product: ovirt-engine
Classification: oVirt
Component: Frontend.WebAdmin (Show other bugs)
3.6.0
Unspecified Unspecified
unspecified Severity medium (vote)
: ---
: ---
Assigned To: Moti Asayag
Pavel Stehlik
infra
: Reopened
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2015-11-14 13:25 EST by Nir Soffer
Modified: 2016-02-10 14:15 EST (History)
7 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2015-12-23 07:40:31 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: Infra
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---
rule-engine: planning_ack?
rule-engine: devel_ack?
rule-engine: testing_ack?


Attachments (Terms of Use)
Engine log (copied about 15 minutes after I saw the error) (1.47 MB, text/plain)
2015-11-14 13:25 EST, Nir Soffer
no flags Details
Server log (14.75 KB, application/x-xz)
2015-11-28 16:34 EST, Nir Soffer
no flags Details
Engine log (71.22 KB, application/x-xz)
2015-11-28 16:34 EST, Nir Soffer
no flags Details

  None (edit)
Description Nir Soffer 2015-11-14 13:25:07 EST
Created attachment 1094126 [details]
Engine log (copied about 15 minutes after I saw the error)

Description of problem:

UI display this error when putting host to maintenance:

Error while executing action: A Request to the Server failed: java.lang.reflect.InvocationTargetException

Host do move to maintenance.

Version-Release number of selected component (if applicable):
oVirt Engine Version: 3.6.0-0.0.master.20150915165908.git51dfb6e.fc22
Browser: Chrome Version 46.0.2490.86 (64-bit)
OS: Fedora 22 (kernel 4.2.5-201.fc22.x86_64)

How reproducible:
Sometimes.

Steps to Reproduce:

This happened several times in the past. This is what I did when it
happened today:

1. Try to migrate vm
2. Migration fails (see bug 1282054)
3. Shutdown vm
4. Remove vm
5. Put non-spm host to maintenance
   Error displayed, host move to maintenance
6. Put spm host to maintenance
   Error displayed, host move to maintenance

Expected results:
Should not display errors when everything seems to work.
Comment 1 Nir Soffer 2015-11-14 14:44:57 EST
Tested again with newer engine:
oVirt Engine Version: 3.6.1-0.0.master.20151113175558.git6a36a6d.fc22

Did same flow, error not displayed. Maybe this was fixed, or maybe something
else is needed to reproduce it.
Comment 2 Allon Mureinik 2015-11-15 03:58:02 EST
Host maintenance is an infra flow. Setting that whiteboard for initial investigation.
Comment 3 Oved Ourfali 2015-11-16 01:15:55 EST
Moti - please take a look.
If reproducible - fix.
If not, close.

Thanks
Comment 4 Moti Asayag 2015-11-16 07:33:31 EST
Tried couple of times - didn't experience the described error message.
Usually additional information for UI related errors could be found on server.log.
Comment 5 Nir Soffer 2015-11-28 16:30:50 EST
Reproduced again with:
oVirt Engine Version: 3.6.1-0.0.master.20151113175558.git6a36a6d.fc22

In a setup with two hosts, select both hosts, click "maintenance", error pops
up, or maybe it pops up after clicking ok.
Comment 6 Nir Soffer 2015-11-28 16:34 EST
Created attachment 1100002 [details]
Server log
Comment 7 Nir Soffer 2015-11-28 16:34 EST
Created attachment 1100003 [details]
Engine log
Comment 8 Nir Soffer 2015-11-28 16:37:30 EST
The error occurred between 7pm and 9pm.
Comment 9 Moti Asayag 2015-11-30 04:15:31 EST
The server.log contains a violation key exception in host-devices table:

Caused by: org.postgresql.util.PSQLException: ERROR: insert or update on table "host_device" violates foreign key constraint "fk_host_device_parent_name"
  Detail: Key (host_id, parent_device_name)=(89887385-48b3-4648-b397-f2078519a61e, scsi_host3) is not present in table "host_device".

The foreign key is defined as: 

SELECT fn_db_create_constraint('host_device', 'fk_host_device_parent_name', 'FOREIGN KEY (host_id, parent_device_name) REFERENCES host_device(host_id, device_name) DEFERRABLE INITIALLY IMMEDIATE');

Martin, could you advise of the above ?
Comment 10 Martin Betak 2015-11-30 07:18:28 EST
It seems the VDSM is reporting inconsistent data for host devices. From the return value of HostDevListByCapsVDSCommand we can see that no 'scsi_host3' device was reported although it is being referenced as parent of 'scsi_target3_0_0' device.

Adding Martin Polednik to investigate.

@Nir: please try to provide relevant VDSM logs if still available.
Comment 11 Martin Polednik 2015-12-01 04:48:27 EST
As Martin requested, we'll need to see the VDSM logs. There shouldn't be any kind of device filtering (except for caps) - VDSM returns devices that libvirt reports. Also, make sure that there is no hook (after_hostdev_list_by_caps) interfering with the call.

It would also be helpful to see 'virsh -r nodedev-list'.
Comment 12 Nir Soffer 2015-12-14 07:11:42 EST
(In reply to Martin Betak from comment #10)
> @Nir: please try to provide relevant VDSM logs if still available.

I don't have the vdsm logs, they were deleted.
Comment 13 Nir Soffer 2015-12-14 07:12:58 EST
(In reply to Martin Polednik from comment #11)
> It would also be helpful to see 'virsh -r nodedev-list'.

I can provide it but not with the vdsm version used in the bug, and I'm not sure
even about rhel/libvirt version, I upgraded this host to 7.2.
Comment 14 Oved Ourfali 2015-12-23 07:40:31 EST
I don't see how we can resolve this without the information requested in the latest comments.
Closing.
If it reproduces, please re-open with all relevant logs, and in addition the output of virsh -r nodedev-list, as Martin requested.

Note You need to log in before you can comment on or make changes to this bug.