Part of the evaluation on this will include whether it needs to be cloned into CP 2.0 as well

I can easily reproduce this on 9.54. After I found out exact cause, I will retest this on 2.0 as well.

This bug is reproducible only on Postgres. It is a timing issue that can be reproduced by concurrently removing content from one product.

Steps to reproduce:
1. Create content POST https://localhost:8443/candlepin/content/
2. Attach the content to a product POST https://localhost:8443/candlepin/products/awesomeos-x86/content/$i
3. In parallel, start deleting content from the product   DELETE https://localhost:8443/candlepin/products/awesomeos-x86/content/$i


The underlying cause for this bug is mapping of ElementCollection [1] on Product entity. The mapping maps @Embeddeable object to List. Since List allows duplicates and Hibernate can't distinguish the objects in the collection in any way (since as @Embeddable they don't have unique @Id), the Hibernate upon every merge of Product entity issues delete of everything from cp_product_content related to the product and then inserts the items from the Product.productContent again [2] (this is known behavior [3] - even though not documented in Hiberante docs). 

Because of the way List and ElementCollection works, the concurrent deletes may interleave in such a fashion that 2 threads T1 and T2 will issue delete statements at the same time and then T1 and T2 inserts the same - duplicit -row.

There are two ways to fix this. First is to use @OrderColumn on he List. However better approach might be just to use Set instead of List for productContent collection. I will issue a PR shortly.

Just a small sidenote to this. Even if we didn't hit this particular timing bug (duplicate key value error) the concurrency in this case HAS BEEN causing other issue on this collection (it just wasn't noticed by anbyody up to know): product content was not being deleted because competing threads were recreating what has just been deleted. 

[1]
 @ElementCollection
    @CollectionTable(name = "cp_product_content",
                     joinColumns = @JoinColumn(name = "product_id"))
    @Column(name = "element")
    @LazyCollection(LazyCollectionOption.EXTRA) // allows .size() without loading all data
    private List<ProductContent> productContent;


[2] 
delete from cp_product_content where product_id=?
insert into cp_product_content (product_id, content_id, enabled, created, updated) values (?, ?, ?, ?, ?)
insert into cp_product_content (product_id, content_id, enabled, created, updated) values (?, ?, ?, ?, ?)
insert into cp_product_content (product_id, content_id, enabled, created, updated) values (?, ?, ?, ?, ?)
....
...

[3] https://en.wikibooks.org/wiki/Java_Persistence/ElementCollection#Primary_keys_in_CollectionTable

There is one more strange constraint violation that I will investigate in this bug 

ERROR: duplicate key value violates unique constraint      "cp_env_cnt_env_id_cntid_ky"

I split up the other issues (cp_env_cnt_env_id_cntid_ky and StaleStateException) to separate BZ's because they are not directly related in Candlepin.