Description of problem: Candlepin is trying to regenerate the SCA certificate(the payload part) when registering a host if there is a content update (CV published new version with new contents) but it didn't delete the existing one before regenerating the new one. Thus, causing the following duplicate key error. ------------------------------------ Caused by: org.postgresql.util.PSQLException: ERROR: duplicate key value violates unique constraint "cp_owner_env_content_access_ukey" ------------------------------------ Version-Release number of selected component (if applicable): Satellite 6.8.5 (Candlepin 3.1.26-1) Steps to Reproduce: 1. Satellite is already running in SCA mode and has some hosts consuming a CV. 2. Unregister 1 of the host 3. Add a repo to the CV and publish a new version. 4. Re-register the host in step (2) to the published CV using AK. subscription-manager register --org="my_org" --activationkey="my_key" Actual results: The system has been registered with ID: <some uuid> The registered system name is: example.host.com Traceback (most recent call last): File "/usr/sbin/subscription-manager", line 9, in <module> load_entry_point('subscription-manager==1.24.45', 'console_scripts', 'subscription-manager')() File "/usr/lib64/python2.7/site-packages/subscription_manager/scripts/subscription_manager.py", line 86, in main return managercli.ManagerCLI().main() File "/usr/lib64/python2.7/site-packages/subscription_manager/managercli.py", line 3065, in main ret = CLI.main(self) File "/usr/lib64/python2.7/site-packages/subscription_manager/cli.py", line 183, in main return cmd.main() File "/usr/lib64/python2.7/site-packages/subscription_manager/managercli.py", line 545, in main return_code = self._do_command() File "/usr/lib64/python2.7/site-packages/subscription_manager/managercli.py", line 1448, in _do_command self.entcertlib.update() File "/usr/lib64/python2.7/site-packages/subscription_manager/certlib.py", line 32, in update self.report = self.locker.run(self._do_update) File "/usr/lib64/python2.7/site-packages/subscription_manager/certlib.py", line 18, in run return action() File "/usr/lib64/python2.7/site-packages/subscription_manager/entcertlib.py", line 42, in _do_update return action.perform() File "/usr/lib64/python2.7/site-packages/subscription_manager/entcertlib.py", line 129, in perform self.install(missing_serials) File "/usr/lib64/python2.7/site-packages/subscription_manager/entcertlib.py", line 178, in install cert_bundles = self.get_certificates_by_serial_list(missing_serials) File "/usr/lib64/python2.7/site-packages/subscription_manager/entcertlib.py", line 285, in get_certificates_by_serial_list serials=sn_list) File "/usr/lib64/python2.7/site-packages/rhsm/connection.py", line 1297, in getCertificates return self.conn.request_get(method) File "/usr/lib64/python2.7/site-packages/rhsm/connection.py", line 729, in request_get return self._request("GET", method, headers=headers) File "/usr/lib64/python2.7/site-packages/rhsm/connection.py", line 755, in _request info=info, headers=headers) File "/usr/lib64/python2.7/site-packages/rhsm/connection.py", line 630, in _request self.validateResponse(result, request_type, handler) File "/usr/lib64/python2.7/site-packages/rhsm/connection.py", line 716, in validateResponse raise NetworkException(response['status']) rhsm.connection.NetworkException: HTTP error (400 - Bad Request) Expected results: No error Additional info: subscription-manager refresh Network error. Please check the connection details, or see /var/log/rhsm/rhsm.log for more information. 2021-04-14 15:15:23,892 [ERROR] subscription-manager:10995:MainThread @managercli.py:217 - Unable to perform refresh due to the following exception: HTTP error (400 - Bad Request) 2021-04-14 15:15:23,892 [ERROR] subscription-manager:10995:MainThread @managercli.py:218 - HTTP error (400 - Bad Request) Candlepin log --------------------------------------------- 2021-04-14 15:46:38,254 [thread=http-bio-127.0.0.1-8443-exec-8] [req=f8e0e91c-74ed-4b70-8542-318414fe73d3, org=hao_org, csid=] INFO org.candlepin.controller.ContentAccessManager - Generating new SCA certificate for organization "hao_org"... 2021-04-14 15:46:38,283 [thread=http-bio-127.0.0.1-8443-exec-8] [req=f8e0e91c-74ed-4b70-8542-318414fe73d3, org=hao_org, csid=] INFO org.candlepin.controller.ContentAccessManager - Generating new SCA payload for organization "hao_org"... 2021-04-14 15:46:39,975 [thread=http-bio-127.0.0.1-8443-exec-8] [req=f8e0e91c-74ed-4b70-8542-318414fe73d3, org=hao_org, csid=] WARN org.hibernate.engine.jdbc.spi.SqlExceptionHelper - SQL Error: 0, SQLState: 23505 2021-04-14 15:46:39,976 [thread=http-bio-127.0.0.1-8443-exec-8] [req=f8e0e91c-74ed-4b70-8542-318414fe73d3, org=hao_org, csid=] ERROR org.hibernate.engine.jdbc.spi.SqlExceptionHelper - ERROR: duplicate key value violates unique constraint "cp_owner_env_content_access_ukey" Detail: Key (owner_id, environment_id)=(8ac705eb723ea1d501726ddf1e110001, 814fd1ec7660730db80f9ca4b9a22a98) already exists. 2021-04-14 15:46:39,983 [thread=http-bio-127.0.0.1-8443-exec-8] [req=f8e0e91c-74ed-4b70-8542-318414fe73d3, org=hao_org, csid=] ERROR org.candlepin.common.exceptions.mappers.CandlepinExceptionMapper - Runtime Error Error while committing the transaction at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse:2,433 javax.persistence.RollbackException: Error while committing the transaction at org.hibernate.internal.ExceptionConverterImpl.convertCommitException(ExceptionConverterImpl.java:81) at org.hibernate.engine.transaction.internal.TransactionImpl.commit(TransactionImpl.java:104) at com.google.inject.persist.jpa.JpaLocalTxnInterceptor.invoke(JpaLocalTxnInterceptor.java:91) at org.candlepin.resource.ConsumerResource.getEntitlementCertificateSerials(ConsumerResource.java:1978) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:151) at org.jboss.resteasy.core.MethodInjectorImpl.lambda$invoke$3(MethodInjectorImpl.java:122) at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:616) at java.util.concurrent.CompletableFuture.uniApplyStage(CompletableFuture.java:628) at java.util.concurrent.CompletableFuture.thenApply(CompletableFuture.java:1996) at java.util.concurrent.CompletableFuture.thenApply(CompletableFuture.java:110) at org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:122) at org.jboss.resteasy.core.ResourceMethodInvoker.internalInvokeOnTarget(ResourceMethodInvoker.java:594) at org.jboss.resteasy.core.ResourceMethodInvoker.invokeOnTargetAfterFilter(ResourceMethodInvoker.java:468) at org.jboss.resteasy.core.ResourceMethodInvoker.lambda$invokeOnTarget$2(ResourceMethodInvoker.java:421) at org.jboss.resteasy.core.interception.jaxrs.PreMatchContainerRequestContext.filter(PreMatchContainerRequestContext.java:363) at org.jboss.resteasy.core.ResourceMethodInvoker.invokeOnTarget(ResourceMethodInvoker.java:423) at org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:391) at org.jboss.resteasy.core.ResourceMethodInvoker.lambda$invoke$1(ResourceMethodInvoker.java:365) at java.util.concurrent.CompletableFuture.uniComposeStage(CompletableFuture.java:995) at java.util.concurrent.CompletableFuture.thenCompose(CompletableFuture.java:2137) at java.util.concurrent.CompletableFuture.thenCompose(CompletableFuture.java:110) at org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:365) at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:477) at org.jboss.resteasy.core.SynchronousDispatcher.lambda$invoke$4(SynchronousDispatcher.java:252) at org.jboss.resteasy.core.SynchronousDispatcher.lambda$preprocess$0(SynchronousDispatcher.java:153) at org.jboss.resteasy.core.interception.jaxrs.PreMatchContainerRequestContext.filter(PreMatchContainerRequestContext.java:363) at org.jboss.resteasy.core.SynchronousDispatcher.preprocess(SynchronousDispatcher.java:156) at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:238) at org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:249) at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:60) at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:55) at javax.servlet.http.HttpServlet.service(HttpServlet.java:731) at com.google.inject.servlet.ServletDefinition.doServiceImpl(ServletDefinition.java:286) at com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:276) at com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:181) at com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:91) at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:85) at org.candlepin.servlet.filter.EventFilter.doFilter(EventFilter.java:65) at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82) at org.candlepin.common.filter.LoggingFilter.doFilter(LoggingFilter.java:123) at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82) at org.candlepin.servlet.filter.CandlepinPersistFilter.doFilter(CandlepinPersistFilter.java:48) at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82) at org.candlepin.servlet.filter.CandlepinScopeFilter.doFilter(CandlepinScopeFilter.java:68) at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82) at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:120) at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:135) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:218) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:110) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:498) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:169) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:445) at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1091) at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:637) at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) at java.lang.Thread.run(Thread.java:748) Caused by: javax.persistence.PersistenceException: org.hibernate.exception.ConstraintViolationException: could not execute statement at org.hibernate.internal.ExceptionConverterImpl.convert(ExceptionConverterImpl.java:154) at org.hibernate.internal.ExceptionConverterImpl.convert(ExceptionConverterImpl.java:181) at org.hibernate.internal.ExceptionConverterImpl.convertCommitException(ExceptionConverterImpl.java:65) ... 66 common frames omitted Caused by: org.hibernate.exception.ConstraintViolationException: could not execute statement at org.hibernate.exception.internal.SQLStateConversionDelegate.convert(SQLStateConversionDelegate.java:109) at org.hibernate.exception.internal.StandardSQLExceptionConverter.convert(StandardSQLExceptionConverter.java:42) at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:113) at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:99) at org.hibernate.engine.jdbc.internal.ResultSetReturnImpl.executeUpdate(ResultSetReturnImpl.java:200) at org.hibernate.persister.entity.AbstractEntityPersister.insert(AbstractEntityPersister.java:3208) at org.hibernate.persister.entity.AbstractEntityPersister.insert(AbstractEntityPersister.java:3722) at org.hibernate.action.internal.EntityInsertAction.execute(EntityInsertAction.java:91) at org.hibernate.engine.spi.ActionQueue.executeActions(ActionQueue.java:604) at org.hibernate.engine.spi.ActionQueue.lambda$executeActions$1(ActionQueue.java:478) at java.util.LinkedHashMap.forEach(LinkedHashMap.java:684) at org.hibernate.engine.spi.ActionQueue.executeActions(ActionQueue.java:475) at org.hibernate.event.internal.AbstractFlushingEventListener.performExecutions(AbstractFlushingEventListener.java:348) at org.hibernate.event.internal.DefaultFlushEventListener.onFlush(DefaultFlushEventListener.java:40) at org.hibernate.event.service.internal.EventListenerGroupImpl.fireEventOnEachListener(EventListenerGroupImpl.java:108) at org.hibernate.internal.SessionImpl.doFlush(SessionImpl.java:1344) at org.hibernate.internal.SessionImpl.managedFlush(SessionImpl.java:435) at org.hibernate.internal.SessionImpl.flushBeforeTransactionCompletion(SessionImpl.java:3221) at org.hibernate.internal.SessionImpl.beforeTransactionCompletion(SessionImpl.java:2389) at org.hibernate.engine.jdbc.internal.JdbcCoordinatorImpl.beforeTransactionCompletion(JdbcCoordinatorImpl.java:447) at org.hibernate.resource.transaction.backend.jdbc.internal.JdbcResourceLocalTransactionCoordinatorImpl.beforeCompletionCallback(JdbcResourceLocalTransactionCoordinatorImpl.java:183) at org.hibernate.resource.transaction.backend.jdbc.internal.JdbcResourceLocalTransactionCoordinatorImpl.access$300(JdbcResourceLocalTransactionCoordinatorImpl.java:40) at org.hibernate.resource.transaction.backend.jdbc.internal.JdbcResourceLocalTransactionCoordinatorImpl$TransactionDriverControlImpl.commit(JdbcResourceLocalTransactionCoordinatorImpl.java:281) at org.hibernate.engine.transaction.internal.TransactionImpl.commit(TransactionImpl.java:101) ... 65 common frames omitted Caused by: org.postgresql.util.PSQLException: ERROR: duplicate key value violates unique constraint "cp_owner_env_content_access_ukey" Detail: Key (owner_id, environment_id)=(8ac705eb723ea1d501726ddf1e110001, 814fd1ec7660730db80f9ca4b9a22a98) already exists. at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2433) at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2178) at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:306) at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:441) at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:365) at org.postgresql.jdbc.PgPreparedStatement.executeWithFlags(PgPreparedStatement.java:155) at org.postgresql.jdbc.PgPreparedStatement.executeUpdate(PgPreparedStatement.java:132) at com.mchange.v2.c3p0.impl.NewProxyPreparedStatement.executeUpdate(NewProxyPreparedStatement.java:462) at org.hibernate.engine.jdbc.internal.ResultSetReturnImpl.executeUpdate(ResultSetReturnImpl.java:197) ... 84 common frames omitted ------------------------------------
Just a speculation about potential workaround: remove the _uniquity_ of the index? (i.e. have that index but not with unique constraint)? Just as a temporary workaround, to let registrations to pass and create duplicate entries in the postgres table. Once we have a permanent fix, remove duplicate entries from the table and make the index unique again..?
This hotfix is applicable to both Satellite 6.8.5 and 6.8.6. Engineering is working on an official errata update to this BZ and we expect this early next week. *** Satellite 6.8.5 and 6.8.6 Hotfix Available *** 1) Download https://people.redhat.com/~mmccune/hotfix/candlepin-3.1.28-1.tar.gz to your Satellite 2) verify sha256 sum: # sha256sum candlepin-3.1.28-1.tar.gz 19d76bbe69b80d792214688b8595d8aa57ad273ef074af8b8552f91d3a2e1035 candlepin-3.1.28-1.tar.gz 3) stop services: satellite-maintain service stop 4) Install: tar xvf candlepin-3.1.28-1.tar.gz rpm -Uvh candlepin*.rpm 4) restart: satellite-maintain service start 5) resume operations
Steps to test: - Deploy a Satellite 6.8.6 from the latest snap. - Upload a manifest with Simple Content Access enabled. - Synchronize two repositories. - Create a content view containing one repository and publish the first version. - Create an activation key associated with the content view. - Register a content host using the activation key. - Unregister the content host. - Add the second repository to the content view and publish a new version. - Re-register the content host using the activation key. Expected Results: The content host is successfully re-registered with no errors. `subscription-manager refresh` also runs without error. Actual Results: The content host is successfully re-registered with no errors. `subscription-manager refresh` also runs without error: ~~~ [root@dhcp-2-49 ~]# subscription-manager register --org="Default_Organization" --activationkey="rhel8" The system has been registered with ID: ff35d8d6-d50e-4258-9fcc-43d2bccd27de The registered system name is: dhcp-2-49.vms.sat.rdu2.redhat.com [root@dhcp-2-49 ~]# subscription-manager refresh All local data refreshed ~~~ Verified on Satellite 6.8.6 (candlepin-3.1.28-1.el7sat.noarch).
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Satellite 6.8 Async Bug Fix Update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2021:1261