Created attachment 578120 [details]
relevant catalina log

per jsherrill, this is different than bug #812346, evidenced by the fact that it does not require the creation of environments.

Candlepin is refusing to delete the content as a product is still set to reference it. This might be wrong and Candlepin should cleanup any product references, but it also might be acting correctly and preventing the deletion of something that crosses organizations, a RH product for example.

I guess it's a question of what product was still referencing that content?

I would try to look at the results of some SQL queries after this happens:

select * from cp_product_content where content_id = failedID;

Then match up with cp_product and see which product that is, is it something used across orgs? 

If this is custom content involved, and definitely isn't used in another org, then we need to decide if Katello can/should delete the product first, or if Candlepin should clean up the references automatically. Given the global nature of products though I am tempted to leave the restriction in place and see if we can't get Katello doing product delete prior to content delete.

(bearing in mind content can be shared by multiple products, used by subscriptions in multiple orgs)

some information:  



    candlepin=# select * from cp_product_content where content_id = '861';
    product_id | content_id | enabled | created | updated
    ------------+------------+---------+---------+---------
    RH0103708 | 861 | f | |
    (1 row)
     
     
    candlepin=# select name from cp_product where id = 'RH0103708';
    name
    -----------------------------------------------------------------------
    Red Hat Enterprise Linux Server, Premium (8 sockets) (Up to 4 guests)
    (1 row)

katello fix:

37f0e5a5d77bbe28fb8dd45dd7c5f358de771c9d

no longer deleting content objects from candlepin in the case of a redhat consumer.  We still disassociate the content from the product in all cases.

actually that was the incorrect hash, still waiting on ack.

fixed in katello:

c89ceafb4d6b09e53898596be67cebe23926bd40

QA Verified in latest brewroot - CloudForms System Engine Version: 0.1.311-1.el6_2

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHEA-2012-0665.html