Bug 1011100

Here is another one, but a little different. Only a 'derived' entry exists for this subscription, but in this case a matching subscription does exist from the service layer. When I do a pool refresh the job gets a state of 3 with no exceptions.

I will note that the subscription in the service layer does not seem to have a derived product on it, so perhaps it was removed.


mysql> select * from cp_pool where subscriptionId = 2691073;
+----------------------------------+---------------------+---------------------+---------------+--------------------+----------------+---------------------+-----------+--------------------------------+----------+----------------------+---------------------+----------------+----------------------------------+----------------------+--------------------+---------+-------------+------------------+--------------------+---------------+-------------------+
| id                               | created             | updated             | accountNumber | activeSubscription | contractNumber | endDate             | productId | productName                    | quantity | restrictedToUsername | startDate           | subscriptionId | owner_id                         | sourceEntitlement_id | subscriptionSubKey | version | orderNumber | derivedProductId | derivedProductName | sourceStackId | sourceConsumer_id |
+----------------------------------+---------------------+---------------------+---------------+--------------------+----------------+---------------------+-----------+--------------------------------+----------+----------------------+---------------------+----------------+----------------------------------+----------------------+--------------------+---------+-------------+------------------+--------------------+---------------+-------------------+
| 8a99f98440e442240140e70ecf865d98 | 2013-09-03 23:40:42 | 2013-09-12 18:21:27 | 5206818       |                   | NULL           | 2014-07-24 23:59:59 | SER0406   | Red Hat OpenStack Tech Preview |       -1 | NULL                 | 2013-07-25 00:00:00 | 2691073        | 8a99f9843a7a3a9d013a8c12f1f738cc | NULL                 | derived            |       6 | NULL        | NULL             | NULL               | NULL          | NULL              |
+----------------------------------+---------------------+---------------------+---------------+--------------------+----------------+---------------------+-----------+--------------------------------+----------+----------------------+---------------------+----------------+----------------------------------+----------------------+--------------------+---------+-------------+------------------+--------------------+---------------+-------------------+
1 row in set (0.81 sec)


mysql> select * from cp_owner where id = '8a99f9843a7a3a9d013a8c12f1f738cc';
+----------------------------------+---------------------+---------------------+-------------+---------+--------------+---------------+---------------------+-------------+
| id                               | created             | updated             | displayName | account | parent_owner | contentPrefix | defaultServiceLevel | upstream_id |
+----------------------------------+---------------------+---------------------+-------------+---------+--------------+---------------+---------------------+-------------+
| 8a99f9843a7a3a9d013a8c12f1f738cc | 2012-10-23 01:23:02 | 2012-10-23 01:23:02 | 6752652     | 6752652 | NULL         | NULL          | NULL                | NULL        |
+----------------------------------+---------------------+---------------------+-------------+---------+--------------+---------------+---------------------+-------------+
1 row in set (0.00 sec)


mysql> select * from cp_job where targetId = '6752652' order by created DESC limit 5;
+----------------------------------------------------+---------------------+---------------------+---------------------+-----------------------------------+---------------------+-------+-------------+-----------------+----------+------------+
| id                                                 | created             | updated             | finishTime          | result                            | startTime           | state | jobGroup    | principalName   | targetId | targetType |
+----------------------------------------------------+---------------------+---------------------+---------------------+-----------------------------------+---------------------+-------+-------------+-----------------+----------+------------+
| refresh_pools_6a3e6d18-bea0-4836-bce5-36d403021d85 | 2013-09-24 09:01:11 | 2013-09-24 09:01:11 | 2013-09-24 09:01:11 | Pools refreshed for owner 6752652 | 2013-09-24 09:01:11 |     3 | async group | candlepin_admin | 6752652  |          0 |
| refresh_pools_2e61a004-d7a0-452d-968e-3d8168844d3f | 2013-09-24 09:00:14 | 2013-09-24 09:00:17 | 2013-09-24 09:00:17 | Pools refreshed for owner 6752652 | 2013-09-24 09:00:14 |     3 | async group | candlepin_admin | 6752652  |          0 |
+----------------------------------------------------+---------------------+---------------------+---------------------+-----------------------------------+---------------------+-------+-------------+-----------------+----------+------------+
2 rows in set (0.00 sec)

May or may not be related to https://bugzilla.redhat.com/show_bug.cgi?id=1009536, but they share some symptoms as sometimes a refresh pool on an account with a derived only can put the system into the same state.

Quick query to see how many derived only subscriptions we have.

select id, created, productId, productName, subscriptionId, subscriptionSubKey from cp_pool where subscriptionSubKey = 'derived' and subscriptionId not in (select subscriptionId from cp_pool where subscriptionSubKey = 'master');

4202 rows in set (1.22 sec)

(In reply to Dennis Crissman from comment #3)
> Quick query to see how many derived only subscriptions we have.
> 
> select id, created, productId, productName, subscriptionId,
> subscriptionSubKey from cp_pool where subscriptionSubKey = 'derived' and
> subscriptionId not in (select subscriptionId from cp_pool where
> subscriptionSubKey = 'master');
> 
> 4202 rows in set (1.22 sec)

Ran this same query in production and got an empty set back.

Created attachment 802439 [details]
derived pools in stage w/out master

Fixed in candlepin.git master: 80ce0976512f72451a89f64dfcb826152e24bebb

Also cherry picked to 0.8.28 hotfix branch and built for IT.

Caused by a recent change to stop cleaning up derived pools during refresh when the subscription is gone/expired. This is fine for derived pools tied to entitlements as they are cleaned up during the cleanup of the main pool, however it is not ok for virt bonus pools which are not tied to any specific pool.

Modified to clean up virt bonus pools as well.

Moving to ON_QA, this is stage and production now. However it will probably be impossible for QE to reproduce and we may have to take IT's word on it's effectiveness.

I am no longer able to find a 'derived' pool w/out a 'master'. Closing this ticket.