| Summary: | [6.2.z] Satellite Candlepin: Bulk delete content hosts error | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| Product: | Red Hat Satellite | Reporter: | Francisco Javier Lopez Y Grueber <flg> | ||||||
| Component: | Candlepin | Assignee: | satellite6-bugs <satellite6-bugs> | ||||||
| Status: | CLOSED CURRENTRELEASE | QA Contact: | Katello QA List <katello-qa-list> | ||||||
| Severity: | high | Docs Contact: | |||||||
| Priority: | high | ||||||||
| Version: | 6.1.10 | CC: | bbuckingham, bcourt, bkearney, cduryee, crog, egolov, fcami, fgarciad, flg, jcallaha, mgazdik, mkalyat, rchauhan, schamilt, shisingh, vrjain, wpinheir | ||||||
| Target Milestone: | Unspecified | Keywords: | FieldEngineering, PrioBumpField, Reopened, Triaged | ||||||
| Target Release: | Unused | ||||||||
| Hardware: | x86_64 | ||||||||
| OS: | Linux | ||||||||
| Whiteboard: | |||||||||
| Fixed In Version: | candlepin-0.9.54.26-1 | Doc Type: | If docs needed, set a value | ||||||
| Doc Text: | Story Points: | --- | |||||||
| Clone Of: | |||||||||
| : | 1402117 1402122 1481367 (view as bug list) | Environment: | |||||||
| Last Closed: | 2018-02-28 19:30:55 UTC | Type: | Bug | ||||||
| Regression: | --- | Mount Type: | --- | ||||||
| Documentation: | --- | CRM: | |||||||
| Verified Versions: | Category: | --- | |||||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||||
| Bug Depends On: | 1402117 | ||||||||
| Bug Blocks: | |||||||||
| Attachments: |
|
||||||||
|
Description
Francisco Javier Lopez Y Grueber
2016-10-31 20:48:41 UTC
Full trace:
>
> 2016-10-31 13:57:17,851 [req=9ef736cf-5a29-47ba-81d3-4fcf09e9e665, org=] INFO org.candlepin.common.filter.LoggingFilter - Request: verb=DELETE, uri=/candlepin/consumers/868dfe31-5768-4060-ac38-d5074ba658a8
> 2016-10-31 13:57:17,943 [req=9ef736cf-5a29-47ba-81d3-4fcf09e9e665, org=CommerzbankAG] INFO org.candlepin.controller.CandlepinPoolManager - Batch revoking entitlements: 3
> 2016-10-31 13:57:17,958 [req=9ef736cf-5a29-47ba-81d3-4fcf09e9e665, org=CommerzbankAG] INFO org.candlepin.controller.CandlepinPoolManager - Starting batch delete of pools and entitlements
> 2016-10-31 13:57:17,994 [req=9ef736cf-5a29-47ba-81d3-4fcf09e9e665, org=CommerzbankAG] INFO org.candlepin.controller.CandlepinPoolManager - All deletes flushed successfully
> 2016-10-31 13:57:18,031 [req=9ef736cf-5a29-47ba-81d3-4fcf09e9e665, org=CommerzbankAG] INFO org.candlepin.controller.CandlepinPoolManager - Modifier entitlements done.
> 2016-10-31 13:57:18,031 [req=9ef736cf-5a29-47ba-81d3-4fcf09e9e665, org=CommerzbankAG] INFO org.candlepin.controller.CandlepinPoolManager - Recomputing status for 1 consumers.
> 2016-10-31 13:57:18,078 [req=9ef736cf-5a29-47ba-81d3-4fcf09e9e665, org=CommerzbankAG] INFO org.candlepin.controller.CandlepinPoolManager - All statuses recomputed.
> 2016-10-31 13:57:18,180 [req=9ef736cf-5a29-47ba-81d3-4fcf09e9e665, org=CommerzbankAG] ERROR org.candlepin.common.exceptions.mappers.CandlepinExceptionMapper - Runtime Error ERROR: update or delete on table "cp_consumer" violates foreign key constraint "fk_sourcestack_consumer" on table "cp_pool_source_stack"
> Detail: Key (id)=(148682475755cb28015774e6f18532f8) is still referenced from table "cp_pool_source_stack". at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse:2,157
> org.postgresql.util.PSQLException: ERROR: update or delete on table "cp_consumer" violates foreign key constraint "fk_sourcestack_consumer" on table "cp_pool_source_stack"
> Detail: Key (id)=(148682475755cb28015774e6f18532f8) is still referenced from table "cp_pool_source_stack".
> at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2157) ~[postgresql-jdbc.jar:na]
> at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1886) ~[postgresql-jdbc.jar:na]
> at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:255) ~[postgresql-jdbc.jar:na]
> at org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:555) ~[postgresql-jdbc.jar:na]
> at org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:417) ~[postgresql-jdbc.jar:na]
> at org.postgresql.jdbc2.AbstractJdbc2Statement.executeUpdate(AbstractJdbc2Statement.java:363) ~[postgresql-jdbc.jar:na]
> at com.mchange.v2.c3p0.impl.NewProxyPreparedStatement.executeUpdate(NewProxyPreparedStatement.java:447) ~[c3p0-0.9.1.2.jar:0.9.1.2]
> at org.hibernate.engine.jdbc.internal.ResultSetReturnImpl.executeUpdate(ResultSetReturnImpl.java:133) ~[hibernate-core-4.2.7.SP2-redhat-1.jar:4.2.7.SP2-redhat-1]
> at org.hibernate.engine.jdbc.batch.internal.NonBatchingBatch.addToBatch(NonBatchingBatch.java:58) ~[hibernate-core-4.2.7.SP2-redhat-1.jar:4.2.7.SP2-redhat-1]
> at org.hibernate.persister.entity.AbstractEntityPersister.delete(AbstractEntityPersister.java:3357) ~[hibernate-core-4.2.7.SP2-redhat-1.jar:4.2.7.SP2-redhat-1]
> at org.hibernate.persister.entity.AbstractEntityPersister.delete(AbstractEntityPersister.java:3560) ~[hibernate-core-4.2.7.SP2-redhat-1.jar:4.2.7.SP2-redhat-1]
> at org.hibernate.action.internal.EntityDeleteAction.execute(EntityDeleteAction.java:102) ~[hibernate-core-4.2.7.SP2-redhat-1.jar:4.2.7.SP2-redhat-1]
> at org.hibernate.engine.spi.ActionQueue.execute(ActionQueue.java:393) ~[hibernate-core-4.2.7.SP2-redhat-1.jar:4.2.7.SP2-redhat-1]
> at org.hibernate.engine.spi.ActionQueue.executeActions(ActionQueue.java:385) ~[hibernate-core-4.2.7.SP2-redhat-1.jar:4.2.7.SP2-redhat-1]
> at org.hibernate.engine.spi.ActionQueue.executeActions(ActionQueue.java:308) ~[hibernate-core-4.2.7.SP2-redhat-1.jar:4.2.7.SP2-redhat-1]
> at org.hibernate.event.internal.AbstractFlushingEventListener.performExecutions(AbstractFlushingEventListener.java:339) ~[hibernate-core-4.2.7.SP2-redhat-1.jar:4.2.7.SP2-redhat-1]
> at org.hibernate.event.internal.DefaultFlushEventListener.onFlush(DefaultFlushEventListener.java:52) ~[hibernate-core-4.2.7.SP2-redhat-1.jar:4.2.7.SP2-redhat-1]
> at org.hibernate.internal.SessionImpl.flush(SessionImpl.java:1240) ~[hibernate-core-4.2.7.SP2-redhat-1.jar:4.2.7.SP2-redhat-1]
> at org.hibernate.ejb.AbstractEntityManagerImpl.flush(AbstractEntityManagerImpl.java:996) ~[hibernate-entitymanager-4.2.7.SP2-redhat-1.jar:4.2.7.SP2-redhat-1]
> at org.candlepin.model.AbstractHibernateCurator.flush(AbstractHibernateCurator.java:364) ~[AbstractHibernateCurator.class:na]
> at org.candlepin.model.AbstractHibernateCurator.save(AbstractHibernateCurator.java:359) ~[AbstractHibernateCurator.class:na]
> at org.candlepin.model.AbstractHibernateCurator.create(AbstractHibernateCurator.java:112) ~[AbstractHibernateCurator.class:na]
> at com.google.inject.persist.jpa.JpaLocalTxnInterceptor.invoke(JpaLocalTxnInterceptor.java:58) ~[guice-persist-3.0-redhat-1.jar:3.0-redhat-1]
> at org.candlepin.model.ConsumerCurator.delete(ConsumerCurator.java:109) ~[ConsumerCurator.class:na]
> at com.google.inject.persist.jpa.JpaLocalTxnInterceptor.invoke(JpaLocalTxnInterceptor.java:58) ~[guice-persist-3.0-redhat-1.jar:3.0-redhat-1]
> at org.candlepin.resource.ConsumerResource.deleteConsumer(ConsumerResource.java:1230) ~[ConsumerResource.class:na]
> ...
> at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:314) [tomcat-coyote.jar:7.0.54]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_111]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_111]
> at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) [tomcat-coyote.jar:7.0.54]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_111]
> 2016-10-31 13:57:18,184 [req=9ef736cf-5a29-47ba-81d3-4fcf09e9e665, org=CommerzbankAG] INFO org.candlepin.common.filter.LoggingFilter - Response: status=500, content-type="application/json", time=333
>
This affects 6.2 as well, the problem is that the data from cp_pool_source_stack are still pointing to the cp_consumer and they are not being deleted in cascade or in advance with the script. From the cp_pool_source_stack definion:
candlepin=# \d cp_pool_source_stack;
Table "public.cp_pool_source_stack"
Column | Type | Modifiers
-------------------+--------------------------+-----------
id | character varying(32) | not null
sourceconsumer_id | character varying(32) | not null
sourcestackid | character varying(255) | not null
derivedpool_id | character varying(32) | not null
created | timestamp with time zone |
updated | timestamp with time zone |
Indexes:
"cp_pool_source_stack_pkey" PRIMARY KEY, btree (id)
"cp_pool_source_stack_pool_ukey" UNIQUE CONSTRAINT, btree (derivedpool_id)
"cp_pool_source_stack_ukey" UNIQUE CONSTRAINT, btree (sourceconsumer_id, sourcestackid)
"idx_sourcestack_pool_fk" btree (derivedpool_id)
Foreign-key constraints:
"fk_sourcestack_consumer" FOREIGN KEY (sourceconsumer_id) REFERENCES cp_consumer(id)
"fk_sourcestack_pool" FOREIGN KEY (derivedpool_id) REFERENCES cp_pool(id) ON DELETE CASCADE
The foreign key:
"fk_sourcestack_consumer" FOREIGN KEY (sourceconsumer_id) REFERENCES cp_consumer(id)
should be changed to:
"fk_sourcestack_consumer" FOREIGN KEY (sourceconsumer_id) REFERENCES cp_consumer(id) ON DELETE CASCADE
Workaround:
The workaround is practically just enabling cascade delete of the pool stacks which are pointing to the cp_consumer which is being deleted.
katello-service stop
service postgresql start
ALTER TABLE cp_pool_source_stack DROP CONSTRAINT fk_sourcestack_consumer;
ALTER TABLE cp_pool_source_stack ADD CONSTRAINT fk_sourcestack_consumer FOREIGN KEY (sourceconsumer_id) REFERENCES cp_consumer(id) ON DELETE CASCADE;
katello-service start
and re-execute the action. Even though this table is not referenced by any other table, I would recommend to put the database schema into the same state it was before and when it was failing after the delete operation will be done. This can be achieved by executing following steps:
katello-service stop
service postgresql start
ALTER TABLE cp_pool_source_stack DROP CONSTRAINT fk_sourcestack_consumer;
ALTER TABLE cp_pool_source_stack ADD CONSTRAINT fk_sourcestack_consumer FOREIGN KEY (sourceconsumer_id) REFERENCES cp_consumer(id);
katello-service start
Is the original action a bulk delete of hypervisors that are providing host based entitlements and guests that are consuming those entitlements at the same time? Yes, the selected overcloud hosts have been redeployed. During stack delete the were not properly deleted. @ Barnaby Court, Is there any reason for having 2 clones of this bug? Thanks! - RF. @ Rodrigo There are 2 clones of the bug because of the different versions of Satellite/Candlepin that are affected. Unfortunately I was unable to reproduce this issue in our environment. If this comes up again, please reopen this BZ (or open a new one) and provide a dump of the database while the issue persists. Once we have a reproducer, we can investigate this further. Fixed upstream in candlepin-0.9.54.26-1 This is fixed in 6.3 GA. I've tested content host removal at 1, 100, and 1000 host intervals. See attached screenshot for verification. Created attachment 1402092 [details]
verification screenshot 1
|