Bug 1766345 - hypervisor checkin fails with cp_consumer_hypervisor_ukey error
Summary: hypervisor checkin fails with cp_consumer_hypervisor_ukey error
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Candlepin
Classification: Community
Component: hypervisor_checkin
Version: 2.9
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: 2.9
Assignee: candlepin-bugs
QA Contact:
URL:
Whiteboard:
Depends On:
Blocks: 1774236 1788671 1788672 1806713
TreeView+ depends on / blocked
 
Reported: 2019-10-28 19:40 UTC by Barnaby Court
Modified: 2023-09-07 20:53 UTC (History)
13 users (show)

Fixed In Version: candlepin-2.9.19-1
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1774236 1788671 1788672 1806713 (view as bug list)
Environment:
Last Closed: 2020-04-23 12:27:37 UTC
Embargoed:


Attachments (Terms of Use)
latest virt-who debug output (1.55 MB, text/plain)
2019-10-31 14:20 UTC, Ramesh kumar
no flags Details
Bug-1766345-Data-Nov-8 (184.81 KB, application/x-xz)
2019-11-08 15:14 UTC, Akash
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Github candlepin candlepin pull 2501 0 'None' closed 1766345: hypervisor checkin fails with cp_consumer_hypervisor_ukey error [ENT-1767] 2021-01-27 23:30:31 UTC

Internal Links: 1788652 1791504

Description Barnaby Court 2019-10-28 19:40:07 UTC
When processing a hypervisor checkin candlepin errors out with a Duplicate entry 'xxxxxxxxx-8a85f981483ab18501484267224f7a0d' for key 'cp_consumer_hypervisor_ukey'

I obfuscated that entry a little bit but many more like it can be found using a splunk search of `candlepin_hosts(env=prod)`  'cp_consumer_hypervisor_ukey'



 Full stack trace is: 

2019-10-28 18:32:44.740+0000 [thread=ITClusteredScheduler_Worker-2] [job=hypervisor_update_19d0696e-1eef-4835-8571-67c79fe1189a, org=XXXXX, csid=] ERROR org.candlepin.pinsetter.core.PinsetterJobListener - Job [hypervisor_update_19d0696e-1eef-4835-8571-67c79fe1189a] failed.
org.quartz.JobExecutionException: Error while committing the transaction
        at org.candlepin.pinsetter.tasks.HypervisorUpdateJob.toExecute(HypervisorUpdateJob.java:194)
        at org.candlepin.pinsetter.tasks.KingpinJob.execute(KingpinJob.java:114)
        at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
        at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573)
Caused by: javax.persistence.RollbackException: Error while committing the transaction
        at org.hibernate.internal.ExceptionConverterImpl.convertCommitException(ExceptionConverterImpl.java:81)
        at org.hibernate.engine.transaction.internal.TransactionImpl.commit(TransactionImpl.java:107)
        at com.google.inject.persist.jpa.JpaLocalTxnInterceptor.invoke(JpaLocalTxnInterceptor.java:91)
        at org.candlepin.pinsetter.tasks.HypervisorUpdateJob.toExecute(HypervisorUpdateJob.java:167)
        ... 3 common frames omitted
Caused by: javax.persistence.PersistenceException: org.hibernate.exception.ConstraintViolationException: could not execute statement
        at org.hibernate.internal.ExceptionConverterImpl.convert(ExceptionConverterImpl.java:154)
        at org.hibernate.internal.ExceptionConverterImpl.convert(ExceptionConverterImpl.java:181)
        at org.hibernate.internal.ExceptionConverterImpl.convertCommitException(ExceptionConverterImpl.java:65)
        ... 6 common frames omitted
Caused by: org.hibernate.exception.ConstraintViolationException: could not execute statement
        at org.hibernate.exception.internal.SQLExceptionTypeDelegate.convert(SQLExceptionTypeDelegate.java:59)
        at org.hibernate.exception.internal.StandardSQLExceptionConverter.convert(StandardSQLExceptionConverter.java:42)
        at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:113)
        at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:99)
        at org.hibernate.engine.jdbc.internal.ResultSetReturnImpl.executeUpdate(ResultSetReturnImpl.java:178)
        at org.hibernate.persister.entity.AbstractEntityPersister.update(AbstractEntityPersister.java:3358)
        at org.hibernate.persister.entity.AbstractEntityPersister.updateOrInsert(AbstractEntityPersister.java:3231)
        at org.hibernate.persister.entity.AbstractEntityPersister.update(AbstractEntityPersister.java:3632)
        at org.hibernate.action.internal.EntityUpdateAction.execute(EntityUpdateAction.java:146)
        at org.hibernate.engine.spi.ActionQueue.executeActions(ActionQueue.java:604)
        at org.hibernate.engine.spi.ActionQueue.executeActions(ActionQueue.java:478)
        at org.hibernate.event.internal.AbstractFlushingEventListener.performExecutions(AbstractFlushingEventListener.java:356)
        at org.hibernate.event.internal.DefaultFlushEventListener.onFlush(DefaultFlushEventListener.java:39)
        at org.hibernate.internal.SessionImpl.doFlush(SessionImpl.java:1454)
        at org.hibernate.internal.SessionImpl.managedFlush(SessionImpl.java:511)
        at org.hibernate.internal.SessionImpl.flushBeforeTransactionCompletion(SessionImpl.java:3283)
        at org.hibernate.internal.SessionImpl.beforeTransactionCompletion(SessionImpl.java:2479)
        at org.hibernate.engine.jdbc.internal.JdbcCoordinatorImpl.beforeTransactionCompletion(JdbcCoordinatorImpl.java:473)
        at org.hibernate.resource.transaction.backend.jdbc.internal.JdbcResourceLocalTransactionCoordinatorImpl.beforeCompletionCallback(JdbcResourceLocalTransactionCoordinatorImpl.java:178)
        at org.hibernate.resource.transaction.backend.jdbc.internal.JdbcResourceLocalTransactionCoordinatorImpl.access$300(JdbcResourceLocalTransactionCoordinatorImpl.java:39)
        at org.hibernate.resource.transaction.backend.jdbc.internal.JdbcResourceLocalTransactionCoordinatorImpl$TransactionDriverControlImpl.commit(JdbcResourceLocalTransactionCoordinatorImpl.java:271)
        at org.hibernate.engine.transaction.internal.TransactionImpl.commit(TransactionImpl.java:104)
        ... 5 common frames omitted
Caused by: com.mysql.jdbc.exceptions.jdbc4.MySQLIntegrityConstraintViolationException: Duplicate entry '-8a85f981483ab18501484267224f7a0d' for key 'cp_consumer_hypervisor_ukey'
        at sun.reflect.GeneratedConstructorAccessor202.newInstance(Unknown Source)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
        at com.mysql.jdbc.Util.getInstance(Util.java:386)
        at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1041)
        at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4190)
        at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4122)
        at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2570)
        at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2731)
        at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2818)
        at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:2157)
        at com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2460)
        at com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2377)
        at com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2361)
        at com.mchange.v2.c3p0.impl.NewProxyPreparedStatement.executeUpdate(NewProxyPreparedStatement.java:462)
        at org.hibernate.engine.jdbc.internal.ResultSetReturnImpl.executeUpdate(ResultSetReturnImpl.java:175)
        ... 22 common frames omitted

Comment 1 William Poteat 2019-10-29 19:03:06 UTC
Can you please pass along the version of virt-who that is being used by the org that results in this error?

Comment 2 Akash 2019-10-30 07:30:03 UTC
Hi William, 
Customer is using latest version of the virt-who package as below, 
virt-who-0.24.7-1.el7.noarch   

Regards,
Akash

Comment 4 Ramesh kumar 2019-10-31 14:20:29 UTC
Created attachment 1631066 [details]
latest virt-who debug output

Hello,

As requested, latest output of virt-who debug execution is attached to Bugzilla. File name: debugvirt-who.txt.

Kindly check and assist. 

Thanks.

Comment 5 Akash 2019-11-08 15:14:22 UTC
Created attachment 1634047 [details]
Bug-1766345-Data-Nov-8

1. virt-who debug from the customer
2. Screenshot from the vCenter for Guest-Host mapping.
3. Screenshot from the Customer Portal for this Machine named 'cbanssbapp1'

Comment 13 William Poteat 2020-01-22 14:21:13 UTC
Current logs do not show the above trace for this org since 1/13/20. The fix for this bug is available on Production already.

Please file as a new bug and include any candelpin logging that might assist in determining the cause of the problem.

Comment 21 Rehana 2020-02-25 07:11:22 UTC
setting needinfo on ramesh for comment 20


Note You need to log in before you can comment on or make changes to this bug.