Bug 1576134

Summary: Failed to remove host xxxxxxxx
Product: Red Hat Enterprise Virtualization Manager Reporter: Ribu Tho <rabraham>
Component: ovirt-engineAssignee: Arik <ahadas>
Status: CLOSED ERRATA QA Contact: Vitalii Yerys <vyerys>
Severity: medium Docs Contact:
Priority: medium    
Version: 4.1.11CC: ahadas, lsurette, lsvaty, michal.skrivanek, mtessun, rabraham, Rhev-m-bugs, srevivo
Target Milestone: ovirt-4.3.0   
Target Release: 4.3.0   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: ovirt-engine-4.3.0_alpha Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-05-08 12:37:35 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: Virt RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Ribu Tho 2018-05-08 23:56:05 UTC
Description of problem:
Unable to remove host from rhevm GUI and got "Failed to remove host xxxxxxxx" message in the event log.

Version-Release number of selected component (if applicable):
ovirt-engine-4.1.11.2-0.1.el7.noarch

How reproducible:
Not always 
Issue is seen in the previous version of 3.3 as per bug https://bugzilla.redhat.com/show_bug.cgi?id=1112359

Steps to Reproduce:
1. Place the host in maintenance mode
2. Try to remove host and reports error as follows

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2018-05-08 12:17:34,777-04 ERROR [org.ovirt.engine.core.bll.RemoveVdsCommand] (org.ovirt.thread.pool-6-thread-30) [283494b5-f10c-467a-a9fb-386c69a8bd68] Exception: org.springframework.dao.DataIntegrityViolationException: CallableStatementCallback; SQL [{call deletevdsstatic(?)}]; ERROR: update or delete on table "vds_static" violates foreign key constraint "vds_static_vm_dynamic_m" on table "vm_dynamic"
  Detail: Key (vds_id)=(c83ddcf6-7d1d-4456-8c7a-667337e2ca53) is still referenced from table "vm_dynamic".
  Where: SQL statement "DELETE
        FROM vds_static
        WHERE vds_id = v_vds_id"
PL/pgSQL function deletevdsstatic(uuid) line 11 at SQL statement; nested exception is org.postgresql.util.PSQLException: ERROR: update or delete on table "vds_static" violates foreign key constraint "vds_static_vm_dynamic_m" on table "vm_dynamic"
  Detail: Key (vds_id)=(c83ddcf6-7d1d-4456-8c7a-667337e2ca53) is still referenced from table "vm_dynamic".
  Where: SQL statement "DELETE
        FROM vds_static
        WHERE vds_id = v_vds_id"
PL/pgSQL function deletevdsstatic(uuid) line 11 at SQL statement
        at org.springframework.jdbc.support.SQLErrorCodeSQLExceptionTranslator.doTranslate(SQLErrorCodeSQLExceptionTranslator.java:243) [spring-jdbc.jar:4.2.4.RELEASE]
        at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:73) [spring-jdbc.jar:4.2.4.RELEASE]
        at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:1094) [spring-jdbc.jar:4.2.4.RELEASE]
        at org.springframework.jdbc.core.JdbcTemplate.call(JdbcTemplate.java:1130) [spring-jdbc.jar:4.2.4.RELEASE]
        at org.springframework.jdbc.core.simple.AbstractJdbcCall.executeCallInternal(AbstractJdbcCall.java:405) [spring-jdbc.jar:4.2.4.RELEASE]
        at org.springframework.jdbc.core.simple.AbstractJdbcCall.doExecute(AbstractJdbcCall.java:365) [spring-jdbc.jar:4.2.4.RELEASE]
        at org.springframework.jdbc.core.simple.SimpleJdbcCall.execute(SimpleJdbcCall.java:198) [spring-jdbc.jar:4.2.4.RELEASE]
        at org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeImpl(SimpleJdbcCallsHandler.java:135) [dal.jar:]
        at org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeImpl(SimpleJdbcCallsHandler.java:130) [dal.jar:]
        at org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeModification(SimpleJdbcCallsHandler.java:76) [dal.jar:]
        at org.ovirt.engine.core.dao.VdsStaticDaoImpl.remove(VdsStaticDaoImpl.java:115) [dal.jar:]
        at org.ovirt.engine.core.dao.VdsStaticDaoImpl.remove(VdsStaticDaoImpl.java:25) [dal.jar:]
        at org.ovirt.engine.core.bll.RemoveVdsCommand.removeVdsStaticFromDb(RemoveVdsCommand.java:145) [bll.jar:]
        at org.ovirt.engine.core.bll.RemoveVdsCommand.lambda$executeCommand$0(RemoveVdsCommand.java:80) [bll.jar:]
        at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInNewTransaction(TransactionSupport.java:202) [utils.jar:]
        at org.ovirt.engine.core.bll.RemoveVdsCommand.executeCommand(RemoveVdsCommand.java:77) [bll.jar:]
        at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1255) [bll.jar:]
        at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1395) [bll.jar:]
        at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:2059) [bll.jar:]
        at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:164) [utils.jar:]
        at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:103) [utils.jar:]
        at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1455) [bll.jar:]
        at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:397) [bll.jar:]
        at org.ovirt.engine.core.bll.PrevalidatingMultipleActionsRunner.executeValidatedCommand(PrevalidatingMultipleActionsRunner.java:204) [bll.jar:]
        at org.ovirt.engine.core.bll.PrevalidatingMultipleActionsRunner.runCommands(PrevalidatingMultipleActionsRunner.java:176) [bll.jar:]
        at org.ovirt.engine.core.bll.PrevalidatingMultipleActionsRunner.lambda$invokeCommands$3(PrevalidatingMultipleActionsRunner.java:182) [bll.jar:]
        at org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil$InternalWrapperRunnable.run(ThreadPoolUtil.java:84) [utils.jar:]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [rt.jar:1.8.0_161]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) [rt.jar:1.8.0_161]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [rt.jar:1.8.0_161]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [rt.jar:1.8.0_161]
        at java.lang.Thread.run(Thread.java:748) [rt.jar:1.8.0_161]
Caused by: org.postgresql.util.PSQLException: ERROR: update or delete on table "vds_static" violates foreign key constraint "vds_static_vm_dynamic_m" on table "vm_dynamic"
  Detail: Key (vds_id)=(c83ddcf6-7d1d-4456-8c7a-667337e2ca53) is still referenced from table "vm_dynamic".
  Where: SQL statement "DELETE
        FROM vds_static
        WHERE vds_id = v_vds_id"
PL/pgSQL function deletevdsstatic(uuid) line 11 at SQL statement
        at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2157)
        at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1886)
        at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:255)
        at org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:555)
        at org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:417)
        at org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:410)
        at org.jboss.jca.adapters.jdbc.CachedPreparedStatement.execute(CachedPreparedStatement.java:303)
        at org.jboss.jca.adapters.jdbc.WrappedPreparedStatement.execute(WrappedPreparedStatement.java:442)
        at org.springframework.jdbc.core.JdbcTemplate$6.doInCallableStatement(JdbcTemplate.java:1133) [spring-jdbc.jar:4.2.4.RELEASE]
        at org.springframework.jdbc.core.JdbcTemplate$6.doInCallableStatement(JdbcTemplate.java:1130) [spring-jdbc.jar:4.2.4.RELEASE]
        at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:1078) [spring-jdbc.jar:4.2.4.RELEASE]
        ... 29 more

2018-05-08 12:17:34,784-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-30) [283494b5-f10c-467a-a9fb-386c69a8bd68] EVENT_ID: USER_FAILED_REMOVE_VDS(106), Correlation ID: 283494b5-f10c-467a-a9fb-386c69a8bd68, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to remove Host xxxxxxxxx (User: admin@internal-authz).
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Actual results:
The host fails from being deleted in GUI. 

Expected results:
The host has to be deleted and removed from GUI. 

Additional info:

Comment 4 Michal Skrivanek 2018-05-09 08:10:51 UTC
engine was restarted while the migration was ongoing and it finished before engine started up so it didn't detect the migration

We can remove migrating_to_vds from vm_dynamic altogether or just clear it out on detecting the VM as Up somewhere, or move all VMs to Unknown on engine start to resolve the actual state on first monitoring cycle.

Comment 6 Michal Skrivanek 2018-07-30 12:26:34 UTC
Arik?

Comment 7 Vitalii Yerys 2018-10-26 14:24:33 UTC
Verified upstream by performing host maintenance/remove procedure via UI with more than 50 attempts. Issue did not reproduce.

Engine:
ovirt-engine-4.3.0-0.0.master.20181024083809.git7991369.el7.noarch

Comment 10 errata-xmlrpc 2019-05-08 12:37:35 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2019:1085

Comment 11 Daniel Gur 2019-08-28 13:12:22 UTC
sync2jira

Comment 12 Daniel Gur 2019-08-28 13:16:34 UTC
sync2jira