Created attachment 1178522 [details] Stack trace showing the candlepin error A katello user reported that they were unable to download the debug certificate from the UI. The attached stack trace shows that candlepin thrown an IndexOutOfBounds exception when fetching the certificate. A very similar issue was addressed with this BZ #1259248 but it appears that the fix has not completely addressed the issue. It appears that candlepin is still deleting the debug certificate. More investigation is needed to determine the cause. The version of candlepin running at the time of the exception was candlepin-0.9.54.6-1.el7.noarch. There were no steps to reproduce provided, but the error was triggered by clicking the 'Generate and Download' button. The workaround included in 1259248 can be used to get around the issue, but DB access is required.
BZ #1259248 was caused when a manifest refresh deleted the debug entitlement and then an attempt was made to download it. This issue appears to be caused by the same process, but I was unable to reproduce it with the latest version of 0.9.54.6. 0.9.54.6-1.el7.noarch does contain the fix for BZ #1259248 and I could find no other way for the debug cert to be deleted. After some investigation, I believe that the user's candlepin instance had been bitten by BZ #1259248 before the user updated the candlepin version that contained the fix, leaving the owner (org) data in a bad state. The bug fix for 1259248 does prevent the issue from happening, but it does not fix the issue once it has happened. This can be addressed.
The fix for BZ #1259248 prevents this BZ from re-occurring, but systems already affected by the bug will require some data cleanup in order to put the system back in the correct state. The following patch will clean up the bad data via a liquibase changeset and will allow certs to be generated again. The fix will be applied once candlepin's DB upgrade script is run. FIXED BY: https://github.com/candlepin/candlepin/pull/1303 MANUALLY APPLYING THE PATCH =========================== If upgrading candlepin is not an option, this patch can be applied manually by running the following DB queries against the candlepin database. POSTGRES: delete from cp_consumer where id IN (select c.id from cp_consumer c left outer join cp_entitlement e on e.consumer_id = c.id where c.name = ueber_cert_consumer' and e.id is null); delete from cp_subscription where id in (select s.id from cp_subscription s inner join cp_product p on s.product_id = p.id left outer join cp_pool pool on pool.productid = p.id where p.name LIKE '%_ueber_product' and pool.id is null); delete from cp_content where id in (select c.id from cp_content c inner join cp_product_content pc on pc.content_id=c.id inner join cp_product p on p.id=pc.product_id left outer join cp_pool pool on pool.productid = p.id where p.name LIKE '%_ueber_product' and pool.id is null); delete from cp_product where id in (select p.id from cp_product p left outer join cp_pool pool on pool.productid = p.id where p.name LIKE '%ueber%' and pool.id is null);