Bug 1354623

Summary: Missing ueber cert errors after upgrade
Product: Red Hat Satellite Reporter: Justin Sherrill <jsherril>
Component: CandlepinAssignee: Michael Stead <mstead>
Status: CLOSED ERRATA QA Contact: jcallaha
Severity: high Docs Contact:
Priority: unspecified    
Version: 6.2.0CC: anerurka, aperotti, bbuckingham, bkearney, brubisch, cduryee, cwelton, egolov, ehelms, jcallaha, mmccune, mstead, vdhande, xdmoon, zhunting
Target Milestone: UnspecifiedKeywords: Reopened, Triaged
Target Release: Unused   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: candlepin-0.9.54.15-1 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1370206 1405518 (view as bug list) Environment:
Last Closed: 2017-01-26 15:54:49 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1354633    
Bug Blocks: 1353215, 1385841, 1405518    

Description Justin Sherrill 2016-07-11 18:28:06 UTC
Description of problem:

Upstream user reported an issue fetching the ueber cert (which prevented capsule syncing) on katello 3.0 (similar to sat 6.2).

While debugging it became clear that it was impossible to correct the issue without going into the candlepin postgresql instance directly.

How reproducible:
Undetermined, may be related to https://bugzilla.redhat.com/show_bug.cgi?id=1259248

Steps to Reproduce:
1. On a satellite 6.1 with a broken ueber cert try to fetch the ueber cert

Actual results:
Traceback below


Expected results:
we get a valid ueber cert



Additional info:


2016-07-08 15:05:48,929 [thread=http-8443-13] [req=4db11df1-4ce5-4e10-947a-f57afb826cf9, org=ShoreTel] ERROR org.candlepin.common.exceptions.mappers.CandlepinExceptionMapper - Runtime Error Index: 0, Size: 0 at java.util.ArrayList.rangeCheck:635
java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
        at java.util.ArrayList.rangeCheck(ArrayList.java:635) ~[na:1.7.0_101]
        at java.util.ArrayList.get(ArrayList.java:411) ~[na:1.7.0_101]
        at org.candlepin.resource.OwnerResource.getUeberCertificate(OwnerResource.java:1325) ~[OwnerResource.class:na]
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.7.0_101]
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) ~[na:1.7.0_101]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.7.0_101]
        at java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_101]
        at org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:168) ~[resteasy-jaxrs-2.3.10.Final.jar:na]
        at org.jboss.resteasy.core.ResourceMethod.invokeOnTarget(ResourceMethod.java:269) ~[resteasy-jaxrs-2.3.10.Final.jar:na]
        at org.jboss.resteasy.core.ResourceMethod.invoke(ResourceMethod.java:227) ~[resteasy-jaxrs-2.3.10.Final.jar:na]
        at org.jboss.resteasy.core.ResourceMethod.invoke(ResourceMethod.java:216) ~[resteasy-jaxrs-2.3.10.Final.jar:na]
        at org.jboss.resteasy.core.SynchronousDispatcher.getResponse(SynchronousDispatcher.java:541) [resteasy-jaxrs-2.3.10.Final.jar:na]
        at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:523) [resteasy-jaxrs-2.3.10.Final.jar:na]
        at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:125) [resteasy-jaxrs-2.3.10.Final.jar:na]
        at org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:208) [resteasy-jaxrs-2.3.10.Final.jar:na]
        at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:55) [resteasy-jaxrs-2.3.10.Final.jar:na]
        at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:50) [resteasy-jaxrs-2.3.10.Final.jar:na]
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:723) [tomcat6-servlet-2.5-api-6.0.24.jar:na]
        at com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:263) [guice-servlet-3.0.jar:na]
        at com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:178) [guice-servlet-3.0.jar:na]
        at com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:91) [guice-servlet-3.0.jar:na]
        at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:62) [guice-servlet-3.0.jar:na]
        at org.candlepin.servlet.filter.EventFilter.doFilter(EventFilter.java:63) [EventFilter.class:na]
        at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:163) [guice-servlet-3.0.jar:na]
        at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) [guice-servlet-3.0.jar

Comment 1 Justin Sherrill 2016-07-11 18:29:52 UTC
Note that this isn't seeking to prevent the original issue, but instead make it easier to resolve or possibly be resolved automatically

Comment 3 Michael Stead 2016-08-02 14:00:52 UTC
The fix for BZ #1259248 prevents this BZ from re-occurring, but systems already affected by the bug will require some data cleanup in order to put the system back in the correct state.

The following patch will clean up the bad data via a liquibase changeset and will allow certs to be generated again. The fix will be applied once candlepin's DB upgrade script is run.

FIXED BY:
https://github.com/candlepin/candlepin/pull/1303


MANUALLY APPLYING THE PATCH
===========================
If upgrading candlepin is not an option, this patch can be applied manually by running the following DB queries against the candlepin database.

POSTGRES:

delete from cp_consumer where id IN (select c.id from cp_consumer c left outer join cp_entitlement e on e.consumer_id = c.id where c.name = ueber_cert_consumer' and e.id is null);

delete from cp_subscription where id in (select s.id from cp_subscription s inner join cp_product p on s.product_id = p.id left outer join cp_pool pool on pool.productid = p.id where p.name LIKE '%_ueber_product' and pool.id is null);

delete from cp_content where id in (select c.id from cp_content c inner join cp_product_content pc on pc.content_id=c.id inner join cp_product p on p.id=pc.product_id left outer join cp_pool pool on pool.productid = p.id where p.name LIKE '%_ueber_product' and pool.id is null);

delete from cp_product where id in (select p.id from cp_product p left outer join cp_pool pool on pool.productid = p.id where p.name LIKE '%ueber%' and pool.id is null);

Comment 4 Zach Huntington-Meath 2016-09-21 20:17:04 UTC
What build of Candlepin has the fix to this issue?

Comment 5 Barnaby Court 2016-09-22 19:15:50 UTC
*** Bug 1367874 has been marked as a duplicate of this bug. ***

Comment 7 jcallaha 2016-10-03 00:18:22 UTC
Verified in Satellite 6.2.2 Async

Unable to reproduce this issue with the latest version of satellite.
Started with two systems, one using default, the other using custom certs. Upgraded the systems and had no issues retrieving the ueber certificates.

Comment 8 Michael Stead 2016-10-03 11:58:38 UTC
Fixed in: candlepin-0.9.54.8+

Comment 10 errata-xmlrpc 2016-10-04 06:43:50 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2016:1993

Comment 12 Bryan Kearney 2016-11-22 14:23:56 UTC
Please open a new bug report for this.

Comment 15 anerurka 2016-11-23 15:04:06 UTC
(In reply to Bryan Kearney from comment #12)
> Please open a new bug report for this.

I have opened a new Bugzilla for the same [https://bugzilla.redhat.com/show_bug.cgi?id=1397924]

Regards,
Ameya Nerurkar

Comment 17 Bryan Kearney 2016-11-28 15:45:24 UTC
Will try this again. Please pull in 

candlepin-0.9.54.15-1 for this bug.

Comment 22 jcallaha 2017-01-18 18:11:41 UTC
Before capsule syncs kicked off

 ueber_subs | ueber_ent_certs | ueber_ents | ueber_pools | ueber_content | ueber_products | ueber_consumer 
------------+-----------------+------------+-------------+---------------+----------------+----------------
          0 |               0 |          0 |           0 |             0 |              0 |              0
(1 row)

During and after capsule syncs

 ueber_subs | ueber_ent_certs | ueber_ents | ueber_pools | ueber_content | ueber_products | ueber_consumer 
------------+-----------------+------------+-------------+---------------+----------------+----------------
          1 |               1 |          1 |           1 |             1 |              1 |              1
(1 row)

Comment 23 jcallaha 2017-01-18 18:14:52 UTC
Verified in Satellite 6.2.7 Snap 2, based on the results in #22

Comment 24 Bryan Kearney 2017-01-26 15:54:49 UTC
This was delivered in satellite 6.2.7 (https://access.redhat.com/errata/RHBA-2017:0197)