+++ This bug was initially created as a clone of Bug #2084130 +++ Description of problem: When a manifest that is old enough (1+ year) is imported into satellite, the CertificateCleanupJob fails with a foreign key constraint error. This has no immediate impact, other than the fact that expired certificates and serials stay in the relevant database tables over time (which CertificateCleanupJob is supposed to solve). Version-Release number of selected component (if applicable): satellite 6.10.0 onwards candlepin-4.0.8-1 onwards How reproducible: 100% Steps to Reproduce: 1. Have a Subscription Allocation (manifest) in the customer portal that was created at least 1 year ago (so that it's Identity Certificate is expired). 2. Export/Import the manifest into Satellite 3. Wait until the CertificateCleanupJob runs in Candlepin (should be running at 12:00 every day, but you could alter it's schedule, e.g. by setting candlepin.async.jobs.CertificateCleanupJob.schedule=0 * * * * ? in candlepin.conf and restarting tomcat to make it run every minute). Actual results: The following stack trace is seen in candlepin.log: 2022-04-18 12:00:00,080 [thread=Thread-25536 (ActiveMQ-client-global-threads)] [job=4cc353697fe0e74101803d9de68456da, job_key=CertificateCleanupJob, org=, csid=] WARN org.hibernate.engine.jdbc.spi.SqlExceptionHelper - SQL Error: 0, SQLState: 23503 2022-04-18 12:00:00,081 [thread=Thread-25536 (ActiveMQ-client-global-threads)] [job=4cc353697fe0e74101803d9de68456da, job_key=CertificateCleanupJob, org=, csid=] ERROR org.hibernate.engine.jdbc.spi.SqlExceptionHelper - ERROR: update or delete on table "cp_id_cert" violates foreign key constraint "fk54b0f288a0b39916" on table "cp_upstream_consumer" Detail: Key (id)=(4cc35369617122e3016171cfa631004e) is still referenced from table "cp_upstream_consumer". 2022-04-18 12:00:00,095 [thread=Thread-25536 (ActiveMQ-client-global-threads)] [job=4cc353697fe0e74101803d9de68456da, job_key=CertificateCleanupJob, org=, csid=] ERROR org.candlepin.async.JobManager - Job "CertificateCleanupJob" failed in 55ms javax.persistence.PersistenceException: org.hibernate.exception.ConstraintViolationException: could not execute statement at org.hibernate.internal.ExceptionConverterImpl.convert(ExceptionConverterImpl.java:154) at org.hibernate.internal.ExceptionConverterImpl.convert(ExceptionConverterImpl.java:181) at org.hibernate.query.internal.AbstractProducedQuery.executeUpdate(AbstractProducedQuery.java:1614) at org.candlepin.model.IdentityCertificateCurator.deleteByIds(IdentityCertificateCurator.java:74) at com.google.inject.persist.jpa.JpaLocalTxnInterceptor.invoke(JpaLocalTxnInterceptor.java:56) at org.candlepin.async.tasks.CertificateCleanupJob.cleanupExpiredIdentityCerts(CertificateCleanupJob.java:91) at org.candlepin.async.tasks.CertificateCleanupJob.execute(CertificateCleanupJob.java:70) at com.google.inject.persist.jpa.JpaLocalTxnInterceptor.invoke(JpaLocalTxnInterceptor.java:64) at org.candlepin.async.JobManager.executeJob(JobManager.java:1333) at org.candlepin.async.JobMessageReceiver$MessageListener.handleMessage(JobMessageReceiver.java:394) at org.candlepin.messaging.impl.artemis.ArtemisConsumer$ArtemisMessageForwarder.onMessage(ArtemisConsumer.java:59) at org.apache.activemq.artemis.core.client.impl.ClientConsumerImpl.callOnMessage(ClientConsumerImpl.java:1030) at org.apache.activemq.artemis.core.client.impl.ClientConsumerImpl.access$400(ClientConsumerImpl.java:49) at org.apache.activemq.artemis.core.client.impl.ClientConsumerImpl$Runner.run(ClientConsumerImpl.java:1153) at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:42) at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:31) at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:65) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) Caused by: org.hibernate.exception.ConstraintViolationException: could not execute statement at org.hibernate.exception.internal.SQLStateConversionDelegate.convert(SQLStateConversionDelegate.java:109) at org.hibernate.exception.internal.StandardSQLExceptionConverter.convert(StandardSQLExceptionConverter.java:42) at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:113) at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:99) at org.hibernate.engine.jdbc.internal.ResultSetReturnImpl.executeUpdate(ResultSetReturnImpl.java:200) at org.hibernate.hql.internal.ast.exec.BasicExecutor.doExecute(BasicExecutor.java:100) at org.hibernate.hql.internal.ast.exec.BasicExecutor.execute(BasicExecutor.java:59) at org.hibernate.hql.internal.ast.exec.DeleteExecutor.execute(DeleteExecutor.java:113) at org.hibernate.hql.internal.ast.QueryTranslatorImpl.executeUpdate(QueryTranslatorImpl.java:454) at org.hibernate.engine.query.spi.HQLQueryPlan.performExecuteUpdate(HQLQueryPlan.java:377) at org.hibernate.internal.SessionImpl.executeUpdate(SessionImpl.java:1420) at org.hibernate.query.internal.AbstractProducedQuery.doExecuteUpdate(AbstractProducedQuery.java:1623) at org.hibernate.query.internal.AbstractProducedQuery.executeUpdate(AbstractProducedQuery.java:1605) ... 17 common frames omitted Caused by: org.postgresql.util.PSQLException: ERROR: update or delete on table "cp_id_cert" violates foreign key constraint "fk54b0f288a0b39916" on table "cp_upstream_consumer" Detail: Key (id)=(4cc35369617122e3016171cfa631004e) is still referenced from table "cp_upstream_consumer". at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2433) at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2178) at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:306) at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:441) at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:365) at org.postgresql.jdbc.PgPreparedStatement.executeWithFlags(PgPreparedStatement.java:155) at org.postgresql.jdbc.PgPreparedStatement.executeUpdate(PgPreparedStatement.java:132) at com.mchange.v2.c3p0.impl.NewProxyPreparedStatement.executeUpdate(NewProxyPreparedStatement.java:462) at org.hibernate.engine.jdbc.internal.ResultSetReturnImpl.executeUpdate(ResultSetReturnImpl.java:197) ... 25 common frames omitted Expected results: The CertificateCleanupJob finishes successfully, without errors in the log. Additional info: