Description of problem: some additional output of the katello side on http://projects.theforeman.org/issues/20853 <pre> 2017-09-05 13:47:10,535 [thread=http-bio-8443-exec-6] [req=15af41a0-8786-45fe-afc4-62b4a2ac3cb1, org=ORG, csid=] ERROR org.candlepin.common.exceptions.mappers.CandlepinExceptionMapper - Runtime Error could not execute statement at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse:2,102 org.hibernate.exception.ConstraintViolationException: could not execute statement at org.hibernate.exception.internal.SQLStateConversionDelegate.convert(SQLStateConversionDelegate.java:112) ~[hibernate-core-5.1.1.Final.jar:5.1.1.Final] at org.hibernate.exception.internal.StandardSQLExceptionConverter.convert(StandardSQLExceptionConverter.java:42) ~[hibernate-core-5.1.1.Final.jar:5.1.1.Final] at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:111) ~[hibernate-core-5.1.1.Final.jar:5.1.1.Final] at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:97) ~[hibernate-core-5.1.1.Final.jar:5.1.1.Final] at org.hibernate.engine.jdbc.internal.ResultSetReturnImpl.executeUpdate(ResultSetReturnImpl.java:207) ~[hibernate-core-5.1.1.Final.jar:5.1.1.Final] at org.hibernate.engine.jdbc.batch.internal.NonBatchingBatch.addToBatch(NonBatchingBatch.java:45) ~[hibernate-core-5.1.1.Final.jar:5.1.1.Final] at org.hibernate.persister.entity.AbstractEntityPersister.delete(AbstractEntityPersister.java:3244) ~[hibernate-core-5.1.1.Final.jar:5.1.1.Final] at org.hibernate.persister.entity.AbstractEntityPersister.delete(AbstractEntityPersister.java:3481) ~[hibernate-core-5.1.1.Final.jar:5.1.1.Final] at org.hibernate.action.internal.EntityDeleteAction.execute(EntityDeleteAction.java:98) ~[hibernate-core-5.1.1.Final.jar:5.1.1.Final] at org.hibernate.engine.spi.ActionQueue.executeActions(ActionQueue.java:582) ~[hibernate-core-5.1.1.Final.jar:5.1.1.Final] at org.hibernate.engine.spi.ActionQueue.executeActions(ActionQueue.java:456) ~[hibernate-core-5.1.1.Final.jar:5.1.1.Final] at org.hibernate.event.internal.AbstractFlushingEventListener.performExecutions(AbstractFlushingEventListener.java:337) ~[hibernate-core-5.1.1.Final.jar:5.1.1.Final] at org.hibernate.event.internal.DefaultFlushEventListener.onFlush(DefaultFlushEventListener.java:39) ~[hibernate-core-5.1.1.Final.jar:5.1.1.Final] at org.hibernate.internal.SessionImpl.flush(SessionImpl.java:1295) ~[hibernate-core-5.1.1.Final.jar:5.1.1.Final] at org.hibernate.jpa.spi.AbstractEntityManagerImpl.flush(AbstractEntityManagerImpl.java:1300) ~[hibernate-entitymanager-5.1.1.Final.jar:5.1.1.Final] at org.candlepin.model.AbstractHibernateCurator.flush(AbstractHibernateCurator.java:527) ~[AbstractHibernateCurator.class:na] at org.candlepin.controller.CandlepinPoolManager.revokeEntitlements(CandlepinPoolManager.java:1786) ~[CandlepinPoolManager.class:na] at com.google.inject.persist.jpa.JpaLocalTxnInterceptor.invoke(JpaLocalTxnInterceptor.java:58) ~[guice-persist-3.0.jar:na] at org.candlepin.controller.CandlepinPoolManager.revokeAllEntitlements(CandlepinPoolManager.java:1928) ~[CandlepinPoolManager.class:na] at com.google.inject.persist.jpa.JpaLocalTxnInterceptor.invoke(JpaLocalTxnInterceptor.java:58) ~[guice-persist-3.0.jar:na] at org.candlepin.controller.CandlepinPoolManager.revokeAllEntitlements(CandlepinPoolManager.java:1921) ~[CandlepinPoolManager.class:na] at com.google.inject.persist.jpa.JpaLocalTxnInterceptor.invoke(JpaLocalTxnInterceptor.java:58) ~[guice-persist-3.0.jar:na] at org.candlepin.resource.ConsumerResource.deleteConsumer(ConsumerResource.java:1325) ~[ConsumerResource.class:na] at com.google.inject.persist.jpa.JpaLocalTxnInterceptor.invoke(JpaLocalTxnInterceptor.java:66) ~[guice-persist-3.0.jar:na] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_141] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_141] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_141] at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_141] at org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:137) ~[resteasy-jaxrs-3.0.10.Final.jar:na] at org.jboss.resteasy.core.ResourceMethodInvoker.invokeOnTarget(ResourceMethodInvoker.java:296) ~[resteasy-jaxrs-3.0.10.Final.jar:na] at org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:250) ~[resteasy-jaxrs-3.0.10.Final.jar:na] at org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:237) ~[resteasy-jaxrs-3.0.10.Final.jar:na] at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:356) [resteasy-jaxrs-3.0.10.Final.jar:na] at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:179) [resteasy-jaxrs-3.0.10.Final.jar:na] at org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:220) [resteasy-jaxrs-3.0.10.Final.jar:na] at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:56) [resteasy-jaxrs-3.0.10.Final.jar:na] at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:51) [resteasy-jaxrs-3.0.10.Final.jar:na] at javax.servlet.http.HttpServlet.service(HttpServlet.java:731) [tomcat-servlet-3.0-api.jar:na] at com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:263) [guice-servlet-3.0.jar:na] at com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:178) [guice-servlet-3.0.jar:na] at com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:91) [guice-servlet-3.0.jar:na] at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:62) [guice-servlet-3.0.jar:na] at org.candlepin.servlet.filter.EventFilter.doFilter(EventFilter.java:61) [EventFilter.class:na] at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:163) [guice-servlet-3.0.jar:na] at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) [guice-servlet-3.0.jar:na] at org.candlepin.servlet.filter.ContentTypeHackFilter.doFilter(ContentTypeHackFilter.java:58) [ContentTypeHackFilter.class:na] at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:163) [guice-servlet-3.0.jar:na] at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) [guice-servlet-3.0.jar:na] at org.candlepin.common.filter.LoggingFilter.doFilter(LoggingFilter.java:112) [candlepin-common-2.0.3.jar:na] at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:163) [guice-servlet-3.0.jar:na] at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) [guice-servlet-3.0.jar:na] at org.candlepin.servlet.filter.CandlepinPersistFilter.doFilter(CandlepinPersistFilter.java:48) [CandlepinPersistFilter.class:na] at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:163) [guice-servlet-3.0.jar:na] at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) [guice-servlet-3.0.jar:na] at org.candlepin.servlet.filter.CandlepinScopeFilter.doFilter(CandlepinScopeFilter.java:68) [CandlepinScopeFilter.class:na] at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:163) [guice-servlet-3.0.jar:na] at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) [guice-servlet-3.0.jar:na] at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:118) [guice-servlet-3.0.jar:na] at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:113) [guice-servlet-3.0.jar:na] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) [catalina.jar:7.0.76] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) [catalina.jar:7.0.76] at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:218) [catalina.jar:7.0.76] at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:110) [catalina.jar:7.0.76] at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:506) [catalina.jar:7.0.76] at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:169) [catalina.jar:7.0.76] at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103) [catalina.jar:7.0.76] at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116) [catalina.jar:7.0.76] at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:445) [catalina.jar:7.0.76] at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1087) [tomcat-coyote.jar:7.0.76] at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:637) [tomcat-coyote.jar:7.0.76] at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316) [tomcat-coyote.jar:7.0.76] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_141] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_141] at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) [tomcat-coyote.jar:7.0.76] at java.lang.Thread.run(Thread.java:748) [na:1.8.0_141] Caused by: org.postgresql.util.PSQLException: ERROR: update or delete on table "cp_certificate" violates foreign key constraint "cp_pool_fk4" on table "cp_pool" Detail: Key (id)=(8a2580945d565740015e294d8b516958) is still referenced from table "cp_pool". at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2102) ~[postgresql-9.0-801.jdbc4.jar:na] at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1835) ~[postgresql-9.0-801.jdbc4.jar:na] at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:257) ~[postgresql-9.0-801.jdbc4.jar:na] at org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:500) ~[postgresql-9.0-801.jdbc4.jar:na] at org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:388) ~[postgresql-9.0-801.jdbc4.jar:na] at org.postgresql.jdbc2.AbstractJdbc2Statement.executeUpdate(AbstractJdbc2Statement.java:334) ~[postgresql-9.0-801.jdbc4.jar:na] at com.mchange.v2.c3p0.impl.NewProxyPreparedStatement.executeUpdate(NewProxyPreparedStatement.java:384) ~[c3p0-0.9.5.2.jar:0.9.5.2] at org.hibernate.engine.jdbc.internal.ResultSetReturnImpl.executeUpdate(ResultSetReturnImpl.java:204) ~[hibernate-core-5.1.1.Final.jar:5.1.1.Final] ... 70 common frames omitted 2017-09-05 13:47:10,535 [thread=http-bio-8443-exec-6] [req=15af41a0-8786-45fe-afc4-62b4a2ac3cb1, org=ORG, csid=] WARN org.candlepin.common.resteasy.filter.LinkHeaderResponseFilter - Method marked for pagination, but no page exists in the context. 2017-09-05 13:47:10,535 [thread=http-bio-8443-exec-6] [req=15af41a0-8786-45fe-afc4-62b4a2ac3cb1, org=ORG, csid=] WARN org.candlepin.audit.EventSinkImpl - Rolling back hornetq transaction. </pre> Version-Release number of selected component (if applicable): candlepin-2.0.40-1.el7.noarch (recently upgraded from candlepin-0.9.54.10-1.el7.noarch ) How reproducible: Delete a hypervisor -> tasks errors Steps to Reproduce: 1. 2. 3. Actual results: deleted hypervisor and fk error Expected results: deleted hypervisor and no error Additional info: I'm not 100% sure the hypervisor is actually deleted, but it doesn't show up in katello UI anymore
it only seems to affect hypervisors that had a license before the migration from candlepin-0.9.54.10-1.el7.noarch to candlepin-2.0.40-1.el7.noarch and I also can't remove licenses from the hypervisors. new hypervisors added after the migration seem to work fine, I can add/remove licenses and delete those hypervisors normally.
After a little investigating I think the issue is that the pool points to a certificate but it should not do that. Newly created hypervisors with assigned licenses don't point to a certificate. As a workaround for my system I ran: update cp_pool set certificate_id = NULL where type = 'UNMAPPED_GUEST' ; update cp_pool set certificate_id = NULL where type = 'ENTITLEMENT_DERIVED' ; Would you guys agree with my assesment and my fix? Greetings Klaas
https://github.com/candlepin/candlepin/pull/1714 - tested it and works here, does this need to go into 2.1 aswell?
commit c0d5566f6a54181f55fd95454e1e72110399ca17 Author: Chris Rog <crog> Date: Mon Sep 18 10:30:22 2017 -0400 1488508: Restricted upstream info migration to master pools - The per-org migration task no longer migrates upstream subscription info to all related pools, but instead only updates the master pool for a given subscription For users already running into this issue, the workaround above will work for the specific issue, but it is recommended to apply a more thorough query that will bring the database into a closer state to what it would be had it been migrated with this fix: UPDATE cp_pool SET cdn_id=NULL, certificate_id=NULL, upstream_entitlement_id=NULL, upstream_consumer_id=NULL, upstream_pool_id=NULL WHERE id IN (SELECT ss.pool_id FROM cp2_pool_source_sub ss WHERE ss.subscription_sub_key != 'master'); This will nullify all six fields that are extraneously updated with upstream information on any pool which is not a master pool.