Bug 2084130

Description of problem:
When a manifest that is old enough (1+ year) is imported into satellite, the CertificateCleanupJob fails with a foreign key constraint error.
This has no immediate impact, other than the fact that expired certificates and serials stay in the relevant database tables over time (which CertificateCleanupJob is supposed to solve).

Version-Release number of selected component (if applicable):
satellite 6.10.0 onwards
candlepin-4.0.8-1 onwards

How reproducible:
100%

Steps to Reproduce:
1. Have a Subscription Allocation (manifest) in the customer portal that was created at least 1 year ago (so that it's Identity Certificate is expired).
2. Export/Import the manifest into Satellite
3. Wait until the CertificateCleanupJob runs in Candlepin (should be running at 12:00 every day, but you could alter it's schedule, e.g. by setting
candlepin.async.jobs.CertificateCleanupJob.schedule=0 * * * * ?
in candlepin.conf and restarting tomcat to make it run every minute).

Actual results:

The following stack trace is seen in candlepin.log:
2022-04-18 12:00:00,080 [thread=Thread-25536 (ActiveMQ-client-global-threads)] [job=4cc353697fe0e74101803d9de68456da, job_key=CertificateCleanupJob, org=, csid=] WARN  org.hibernate.engine.jdbc.spi.SqlExceptionHelper - SQL Error: 0, SQLState: 23503
2022-04-18 12:00:00,081 [thread=Thread-25536 (ActiveMQ-client-global-threads)] [job=4cc353697fe0e74101803d9de68456da, job_key=CertificateCleanupJob, org=, csid=] ERROR org.hibernate.engine.jdbc.spi.SqlExceptionHelper - ERROR: update or delete on table "cp_id_cert" violates foreign key constraint "fk54b0f288a0b39916" on table "cp_upstream_consumer"
  Detail: Key (id)=(4cc35369617122e3016171cfa631004e) is still referenced from table "cp_upstream_consumer".
2022-04-18 12:00:00,095 [thread=Thread-25536 (ActiveMQ-client-global-threads)] [job=4cc353697fe0e74101803d9de68456da, job_key=CertificateCleanupJob, org=, csid=] ERROR org.candlepin.async.JobManager - Job "CertificateCleanupJob" failed in 55ms
javax.persistence.PersistenceException: org.hibernate.exception.ConstraintViolationException: could not execute statement
	at org.hibernate.internal.ExceptionConverterImpl.convert(ExceptionConverterImpl.java:154)
	at org.hibernate.internal.ExceptionConverterImpl.convert(ExceptionConverterImpl.java:181)
	at org.hibernate.query.internal.AbstractProducedQuery.executeUpdate(AbstractProducedQuery.java:1614)
	at org.candlepin.model.IdentityCertificateCurator.deleteByIds(IdentityCertificateCurator.java:74)
	at com.google.inject.persist.jpa.JpaLocalTxnInterceptor.invoke(JpaLocalTxnInterceptor.java:56)
	at org.candlepin.async.tasks.CertificateCleanupJob.cleanupExpiredIdentityCerts(CertificateCleanupJob.java:91)
	at org.candlepin.async.tasks.CertificateCleanupJob.execute(CertificateCleanupJob.java:70)
	at com.google.inject.persist.jpa.JpaLocalTxnInterceptor.invoke(JpaLocalTxnInterceptor.java:64)
	at org.candlepin.async.JobManager.executeJob(JobManager.java:1333)
	at org.candlepin.async.JobMessageReceiver$MessageListener.handleMessage(JobMessageReceiver.java:394)
	at org.candlepin.messaging.impl.artemis.ArtemisConsumer$ArtemisMessageForwarder.onMessage(ArtemisConsumer.java:59)
	at org.apache.activemq.artemis.core.client.impl.ClientConsumerImpl.callOnMessage(ClientConsumerImpl.java:1030)
	at org.apache.activemq.artemis.core.client.impl.ClientConsumerImpl.access$400(ClientConsumerImpl.java:49)
	at org.apache.activemq.artemis.core.client.impl.ClientConsumerImpl$Runner.run(ClientConsumerImpl.java:1153)
	at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:42)
	at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:31)
	at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:65)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
Caused by: org.hibernate.exception.ConstraintViolationException: could not execute statement
	at org.hibernate.exception.internal.SQLStateConversionDelegate.convert(SQLStateConversionDelegate.java:109)
	at org.hibernate.exception.internal.StandardSQLExceptionConverter.convert(StandardSQLExceptionConverter.java:42)
	at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:113)
	at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:99)
	at org.hibernate.engine.jdbc.internal.ResultSetReturnImpl.executeUpdate(ResultSetReturnImpl.java:200)
	at org.hibernate.hql.internal.ast.exec.BasicExecutor.doExecute(BasicExecutor.java:100)
	at org.hibernate.hql.internal.ast.exec.BasicExecutor.execute(BasicExecutor.java:59)
	at org.hibernate.hql.internal.ast.exec.DeleteExecutor.execute(DeleteExecutor.java:113)
	at org.hibernate.hql.internal.ast.QueryTranslatorImpl.executeUpdate(QueryTranslatorImpl.java:454)
	at org.hibernate.engine.query.spi.HQLQueryPlan.performExecuteUpdate(HQLQueryPlan.java:377)
	at org.hibernate.internal.SessionImpl.executeUpdate(SessionImpl.java:1420)
	at org.hibernate.query.internal.AbstractProducedQuery.doExecuteUpdate(AbstractProducedQuery.java:1623)
	at org.hibernate.query.internal.AbstractProducedQuery.executeUpdate(AbstractProducedQuery.java:1605)
	... 17 common frames omitted
Caused by: org.postgresql.util.PSQLException: ERROR: update or delete on table "cp_id_cert" violates foreign key constraint "fk54b0f288a0b39916" on table "cp_upstream_consumer"
  Detail: Key (id)=(4cc35369617122e3016171cfa631004e) is still referenced from table "cp_upstream_consumer".
	at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2433)
	at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2178)
	at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:306)
	at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:441)
	at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:365)
	at org.postgresql.jdbc.PgPreparedStatement.executeWithFlags(PgPreparedStatement.java:155)
	at org.postgresql.jdbc.PgPreparedStatement.executeUpdate(PgPreparedStatement.java:132)
	at com.mchange.v2.c3p0.impl.NewProxyPreparedStatement.executeUpdate(NewProxyPreparedStatement.java:462)
	at org.hibernate.engine.jdbc.internal.ResultSetReturnImpl.executeUpdate(ResultSetReturnImpl.java:197)
	... 25 common frames omitted


Expected results:
The CertificateCleanupJob finishes successfully, without errors in the log.

Additional info: