Bugzilla (bugzilla.redhat.com) will be under maintenance for infrastructure upgrades and will not be unavailable on July 31st between 12:30 AM - 05:30 AM UTC. We appreciate your understanding and patience. You can follow status.redhat.com for details.
Bug 1344402 - ERROR: duplicate key value violates unique constraint "cp_product_content_pkey"
Summary: ERROR: duplicate key value violates unique constraint "cp_product_content_pkey"
Alias: None
Product: Candlepin
Classification: Community
Component: candlepin
Version: 0.9.54
Hardware: Unspecified
OS: Unspecified
Target Milestone: ---
: ---
Assignee: Filip Nguyen
QA Contact: Katello QA List
Depends On:
Blocks: 1344257 1345918
TreeView+ depends on / blocked
Reported: 2016-06-09 15:12 UTC by Barnaby Court
Modified: 2016-06-13 22:17 UTC (History)
7 users (show)

Fixed In Version: candlepin-
Doc Type: If docs needed, set a value
Doc Text:
Clone Of: 1344257
: 1345918 (view as bug list)
Last Closed: 2016-06-13 22:17:32 UTC

Attachments (Terms of Use)

System ID Private Priority Status Summary Last Updated
Red Hat Bugzilla 1345765 0 low CLOSED Concurrent content demotions cause 400 2021-02-22 00:41:40 UTC
Red Hat Bugzilla 1345766 0 low CLOSED Concurrent content promotion cause 400 2021-02-22 00:41:40 UTC

Internal Links: 1345765 1345766

Comment 1 Barnaby Court 2016-06-09 15:13:08 UTC
Part of the evaluation on this will include whether it needs to be cloned into CP 2.0 as well

Comment 2 Filip Nguyen 2016-06-10 09:37:30 UTC
I can easily reproduce this on 9.54. After I found out exact cause, I will retest this on 2.0 as well.

Comment 3 Filip Nguyen 2016-06-10 11:17:13 UTC
This bug is reproducible only on Postgres. It is a timing issue that can be reproduced by concurrently removing content from one product.

Steps to reproduce:
1. Create content POST https://localhost:8443/candlepin/content/
2. Attach the content to a product POST https://localhost:8443/candlepin/products/awesomeos-x86/content/$i
3. In parallel, start deleting content from the product   DELETE https://localhost:8443/candlepin/products/awesomeos-x86/content/$i

The underlying cause for this bug is mapping of ElementCollection [1] on Product entity. The mapping maps @Embeddeable object to List. Since List allows duplicates and Hibernate can't distinguish the objects in the collection in any way (since as @Embeddable they don't have unique @Id), the Hibernate upon every merge of Product entity issues delete of everything from cp_product_content related to the product and then inserts the items from the Product.productContent again [2] (this is known behavior [3] - even though not documented in Hiberante docs). 

Because of the way List and ElementCollection works, the concurrent deletes may interleave in such a fashion that 2 threads T1 and T2 will issue delete statements at the same time and then T1 and T2 inserts the same - duplicit -row.

There are two ways to fix this. First is to use @OrderColumn on he List. However better approach might be just to use Set instead of List for productContent collection. I will issue a PR shortly.

Just a small sidenote to this. Even if we didn't hit this particular timing bug (duplicate key value error) the concurrency in this case HAS BEEN causing other issue on this collection (it just wasn't noticed by anbyody up to know): product content was not being deleted because competing threads were recreating what has just been deleted. 

    @CollectionTable(name = "cp_product_content",
                     joinColumns = @JoinColumn(name = "product_id"))
    @Column(name = "element")
    @LazyCollection(LazyCollectionOption.EXTRA) // allows .size() without loading all data
    private List<ProductContent> productContent;

delete from cp_product_content where product_id=?
insert into cp_product_content (product_id, content_id, enabled, created, updated) values (?, ?, ?, ?, ?)
insert into cp_product_content (product_id, content_id, enabled, created, updated) values (?, ?, ?, ?, ?)
insert into cp_product_content (product_id, content_id, enabled, created, updated) values (?, ?, ?, ?, ?)

[3] https://en.wikibooks.org/wiki/Java_Persistence/ElementCollection#Primary_keys_in_CollectionTable

Comment 4 Filip Nguyen 2016-06-13 06:34:48 UTC
There is one more strange constraint violation that I will investigate in this bug 

ERROR: duplicate key value violates unique constraint      "cp_env_cnt_env_id_cntid_ky"

Comment 6 Filip Nguyen 2016-06-13 07:14:59 UTC
I split up the other issues (cp_env_cnt_env_id_cntid_ky and StaleStateException) to separate BZ's because they are not directly related in Candlepin.

Note You need to log in before you can comment on or make changes to this bug.