Bug 671195

Summary: Entitlement Beta subscription quantity falls below 0
Product: [Community] Candlepin Reporter: John Sefler <jsefler>
Component: candlepinAssignee: Jesus M. Rodriguez <jesusr>
Status: CLOSED CURRENTRELEASE QA Contact: John Sefler <jsefler>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 0.5CC: bkearney, dgoodwin, jesusr, msechres
Target Milestone: ---Keywords: QA-Closed
Target Release: ---   
Hardware: Unspecified   
OS: Solaris   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2011-02-23 21:51:42 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 639436    
Attachments:
Description Flags
consuming 2 of 1 from the subscription pool? How'd I do that?
none
remove consumed attr to avoid concurrency issue.
none
Free Entitlements? Bad for business! none

Description John Sefler 2011-01-20 18:05:50 UTC
Description of problem:
At this time I don't know how to reproduce this error, and I'm blocked by https://bugzilla.redhat.com/show_bug.cgi?id=671180 when trying to reproduce it.  Nevertheless...  Somehow I got into a state where my personal entitlement beta subscription quantity decremented below zero.

See the attached screenshot and the -1 quantity in the all available list below

[root@jsefler-betaqa-2 ~]# subscription-manager list --available --all
+-------------------------------------------+
    Available Subscriptions
+-------------------------------------------+


ProductName:       	Red Hat Enterprise Linux Server Entitlement Beta for
                        Certified Engineers and System Administrators - NOT FOR SALE
ProductId:         	RH3016037                
PoolId:            	8a8aa80d2d953ffa012d95ace14b0045
Quantity:          	-1                       
Expires:           	2011-07-16               


ProductName:       	RH3036913                
ProductId:         	RH3036913                
PoolId:            	8a8aa80d2d95d36d012d95eb32110008
Quantity:          	10                       
Expires:           	2011-07-16   



Version-Release number of selected component (if applicable):
[root@jsefler-betaqa-2 ~]# rpm -qa | grep subscription-manager
subscription-manager-gnome-0.93.13-1.git.6.1cac2dd.el6.x86_64
subscription-manager-0.93.13-1.git.6.1cac2dd.el6.x86_64
subscription-manager-firstboot-0.93.13-1.git.6.1cac2dd.el6.x86_64

[root@jsefler-betaqa-2 ~]# grep hostname /etc/rhsm/rhsm.conf 
# Server hostname:
hostname = subscriptions.rhn.webqa.redhat.com
proxy_hostname =




Additional info:
Once I get unblocked and can reproduce, I'll add more details.

Comment 2 John Sefler 2011-01-21 17:08:00 UTC
Created attachment 474643 [details]
consuming 2 of 1 from the subscription pool?  How'd I do that?

Comment 4 Jesus M. Rodriguez 2011-02-03 22:20:02 UTC
I was never able to recreate the case where the parent pool was over consumed
i.e. 2 of 1. But I was able to take the consumed count negative effectively
adding entitlements.

Easiest way to reproduce:

1) start with 3 clients (either real machines or guests)
2) remove the consumer and entitlement certs (just to make sure 
   there's no old stuff hanging around):

   rm -f /etc/pki/entitlement/* /etc/pki/consumer/*

3) register all 3 clients (one at a time)
   subscription-manager register --user USER --pass PASS

4) Find an available pool to subscribe to, make a note of the pool id:

   subscription-manager list --available

5) This is the MOST important step. Type in the subscribe command but 
   do *NOT* hit enter yet. Simply stage the command in each guest:

   subscription-manager subscribe --pool POOLID

   *DO NOT HIT ENTER YET*

6) Now login to the webui to verify that the consumption of the entitlement
  currently is.

7) Now, hit enter in each of your client windows. 

8) Refresh the webui and the consumption should be increased by 3.

Comment 5 Jesus M. Rodriguez 2011-02-03 22:23:49 UTC
If testing against on-premise, you will need to have the debugger attached to tomcat in order to get it to 'slow down' enough for the timing to work.

6) attached the debugger to tomcat

7) connect to psql and run the following query:

   select e.id, e.quantity from cp_entitlement e where id in (select entitlement_id from cp_pool_entitlements where pool_id = 'POOLID_FROM_STEP_5');

   not the number of rows (might be 0 rows)

8) Now, hit enter in each of your client windows.

9) *DISCONNECT* the debugger

10) the clients should subscribe to the pool

11) rerun the query from step 7, notice the rows returned are now 
    increased by 3.

Comment 6 Jesus M. Rodriguez 2011-02-03 22:34:18 UTC
Created attachment 476903 [details]
remove consumed attr to avoid concurrency issue.

Comment 7 Jesus M. Rodriguez 2011-02-04 15:04:52 UTC
requires candlepin-0.1.34-1 or greater

fixed by a40092984eae5044a9f5d7334a252b03fa243380

Comment 8 John Sefler 2011-02-05 00:38:39 UTC
Created attachment 477153 [details]
Free Entitlements?  Bad for business!

ATTEMPT TO VERIFY....
[root@jsefler-betaqa-1 ~]#  curl -u foo:bar -k  https://subscriptions.rhn.webqa.redhat.com:443/subscription/status
{"version":"0.1.34","result":true,"release":"1%{?dist}"}[root@jsefler-betaqa-1 ~]# 

Test Strategy...
After creating a personal consumer against the QA environment, I subscribed 9 consumers as follows...

[root@jsefler-betaqa-1 ~]# for i in `seq 1 9`; do subscription-manager clean; subscription-manager register --username=jsefler-qabetauser-14 --password=redhat; subscription-manager subscribe --pool=8a9b90882df297d5012df31def5e00bb; done

That left me with 9 of 10 used entitlements.

Then on four registered clients, I simultaneously hit enter to the following command:
# subscription-manager subscribe --pool=8a9b90882df297d5012df31def5e00bb


Expected Results:
One of the four clients should succeed and three of them should return:
No free entitlements are available for the pool with id '8a9b90882df297d5012df31def5e00bb'

Actual Results:
All four succeeded resulting in 13 of 10 used entitlements.

See attached screenshot

Moving back to ASSIGNED

Comment 10 Devan Goodwin 2011-02-11 13:15:00 UTC
Fixed by implementing a pessimistic lock on the pool during the bind operation.

master: d1ef209fa67af1a16c102d13dcf69659cbacf3d9, will appear in candlepin-0.2.5

Not yet pushed to other branches, waiting to see how it holds up.

Comment 11 John Sefler 2011-02-23 19:50:23 UTC
Moving to VERIFIED
Comment #8 tests are passing against master candlepin version 0.2.6
Comment #8 tests are passing against BETA candlepin version 0.1.38

Parallel attempts to subscribe beyond the entitlement quantity available are now resulting in success for only the number of free entitlements and rejection for all the other parallel subscribers.