Created attachment 1217382 [details]
A script to create a new owner and concurrently run ueber cert generation.

+++ This bug was initially created as a clone of Bug #1391922 +++

Description of problem:

When concurrent requests to generate a new ueber cert for the same owner occur, a request can fail resulting in bad data in the DB. Once this happens, it prevents clients from re-generating, and fetching the cert.

Candlepin requires better transaction management when generating the cert and should lock the Owner to ensure that only a single cert is created per owner.

How reproducible:
100%

Steps to Reproduce:
1. Start with a fresh candlepin database.
2. Run the attached script to create a new owner, and concurrently run the ueber cert generation API. It may take a few runs of the script to get the bug to occur. Specify a new owner each time.


Actual results:

After running the script above, exceptions are thrown and the ueber cert data in the candlepin database is incomplete, which prevents generating a new cert.


$ ./concurrent_ueber_cert.sh -o mtest
Creating owner mtest and generating certs...
{
  "parentOwner" : null,
  "id" : "402882e7582b929701582f06a6a32b18",
  "key" : "mtest",
  "displayName" : "mtest",
  "contentPrefix" : null,
  "defaultServiceLevel" : null,
  "upstreamConsumer" : null,
  "logLevel" : null,
  "autobindDisabled" : null,
  "href" : "/owners/mtest",
  "created" : "2016-11-04T11:08:34.851+0000",
  "updated" : "2016-11-04T11:08:34.851+0000"
}[mstead@bluestar temp]$ {
{
  "displayMessage" : "Problem generating ueber cert for owner org.hibernate.NonUniqueResultException: query did not return a unique result: 2",
  "displayMessage" : "Problem generating ueber cert for owner org.hibernate.NonUniqueResultException: query did not return a unique result: 2",
  "requestUuid" : "0ad932f6-08e8-4e9e-9d35-58890040ac64"
  "requestUuid" : "60a07366-ea56-40d0-8742-a28c48542dc9"
}}

As shown in the results, the two requests collide causing hibernate exceptions. The resulting data (below) shows partial commits to the DB. The query should show a single set of ueber cert data per Org.


candlepin=# SELECT (SELECT Count(*)
        FROM   cp_subscription s
               INNER JOIN cp_product p
                       ON s.product_id = p.id
               LEFT OUTER JOIN cp_pool pool
                            ON pool.productid = p.id
        WHERE  p.NAME LIKE '%_ueber_product')    AS UEBER_SUBS,
       (SELECT Count(*)
        FROM   cp_ent_certificate ec
               INNER JOIN cp_entitlement e
                       ON e.id = ec.entitlement_id
               INNER JOIN cp_pool p
                       ON e.pool_id = p.id
               INNER JOIN cp_product prod
                       ON prod.id = p.productid
        WHERE  prod.NAME LIKE '%_ueber_product') AS UEBER_ENT_CERTS,
       (SELECT Count(*)
        FROM   cp_entitlement e
               INNER JOIN cp_pool p
                       ON e.pool_id = p.id
               INNER JOIN cp_product prod
                       ON prod.id = p.productid
        WHERE  prod.NAME LIKE '%_ueber_product') AS UEBER_ENTS,
        WHERE  NAME = 'ueber_cert_consumer')     AS UEBER_CONSUMER;  
 ueber_subs | ueber_ent_certs | ueber_ents | ueber_pools | ueber_content | ueber_products 
| ueber_consumer 
------------+-----------------+------------+-------------+---------------+----------------
+----------------
          2 |               0 |          0 |           2 |             2 |              2 
|              2
(1 row)

NOTE: The query result should have been 1 for each column.


Expected results:

After the above errors, we are unable to get the ueber cert for the target owner.

$ curl -k -u admin:admin https://localhost:8443/candlepin/owners/mtest/uebercert
{
  "displayMessage" : "Runtime Error query did not return a unique result: 2 at org.hibernate.internal.AbstractQueryImpl.uniqueElement:914",
  "requestUuid" : "dc9fdfbd-8707-4b53-ae51-d25952352d78"
}


We are also unable to generate a new one:

$ curl -k -u admin:admin -X POST https://localhost:8443/candlepin/owners/mtest/uebercert
{
  "displayMessage" : "Problem generating ueber cert for owner org.hibernate.NonUniqueResultException: query did not return a unique result: 2",
  "requestUuid" : "bac32cb7-9497-4eb0-832d-775fd1b9efa9"
}

Expected results:

Concurrent requests should succeed, and result in a single ueber certificate per Owner. The DB query above should show an equal count X per column, where X is the number of Owners that have an ueber cert generated.

Clients should be able to both fetch an existing cert and generate a new cert. 


Additional info:

NOTE: It may take a few runs of the script to get the bug to occur. Specify a new owner each time with -o NEW_OWNER_KEY