Description of problem:
Observed on a client (which is 5min behind the EST time zone) a inactive "golden ticket" entitlement was attached. After auto-attach was run the certificate was replaced by a active entitlement having same serial number

Version-Release number of selected component (if applicable):
subscription management server: 2.0.25-1
subscription management rules: 5.20
subscription-manager: 1.19.1-1.git.9.908316a.el6
python-rhsm: 1.19.1-1.git.3.9985682.el6


How reproducible:
always

Steps to Reproduce:
1.setup 2 client test machines : Machine 1 in EST timezone, Machine 2 in EST -5mins 
2.Register  Machine 1 against "snowwhite" ==> observed that an active "golden ticket" entitlement is attached
# date
Fri Feb 17 07:39:56 EST 2017   ============>> NOTICE the time pls 

[root@dhcp35-181 ~]# subscription-manager list --consumed
+-------------------------------------------+
   Consumed Subscriptions
+-------------------------------------------+
Subscription Name:   Awesome OS Instance Based (Standard Support)
Provides:            Awesome OS Instance Server Bits
SKU:                 awesomeos-instancebased
Contract:            1
Account:             12331131231
Serial:              222045750908889736
Pool ID:             8ac6a3635a3749c0015a374ad37e0af9
Provides Management: No
Active:              True
Quantity Used:       2
Service Level:       Standard
Service Type:        L1-L3
Status Details:      Subscription is current
Subscription Type:   Instance Based
Starts:              02/12/2017
Ends:                02/12/2018
System Type:         Physical

Subscription Name:   Content Access
Provides:             Content Access
SKU:                 content_access
Contract:            
Account:             
Serial:              7130814772046333282  
Pool ID:             Not Available
Provides Management: No
Active:              True
Quantity Used:       1
Service Level:       
Service Type:        
Status Details:   Subscription is current => golden ticket entitlement is active
Subscription Type:   
Starts:              02/17/2017
Ends:                02/17/2018
System Type:         Physical


3.Register Machine 2 against "snowwhite" ==> observed a FUTURE "golden ticket" is being attached.

[root@dhcp35-231 facts]# date
Fri Feb 17 07:34:13 EST 2017  =======>> Notice the difference in mins , -5 min behind the 1st machine 

[root@dhcp35-231 facts]# ll /etc/pki/entitlement/
total 16
-rw-------. 1 root root 1675 Feb 17 07:33 7010439351232508225-key.pem
-rw-r--r--. 1 root root 2449 Feb 17 07:33 7010439351232508225.pem
-rw-------. 1 root root 1675 Feb 17 07:33 842082998312036743-key.pem
-rw-r--r--. 1 root root 2878 Feb 17 07:33 842082998312036743.pem
[root@dhcp35-231 facts]# subscription-manager list --consumed
+-------------------------------------------+
   Consumed Subscriptions
+-------------------------------------------+
Subscription Name:   Content Access
Provides:             Content Access
SKU:                 content_access
Contract:            
Account:             
Serial:              842082998312036743
Pool ID:             Not Available
Provides Management: No
Active:              False
Quantity Used:       1
Service Level:       
Service Type:        
Status Details:      Subscription has not begun  ==============> says its a future subscription
Subscription Type:   
Starts:              02/17/2017
Ends:                02/17/2018
System Type:         Physical

Subscription Name:   Awesome OS Instance Based (Standard Support)
Provides:            Awesome OS Instance Server Bits
SKU:                 awesomeos-instancebased
Contract:            1
Account:             12331131231
Serial:              7010439351232508225
Pool ID:             8ac6a3635a3749c0015a374ad37e0af9
Provides Management: No
Active:              True
Quantity Used:       2
Service Level:       Standard
Service Type:        L1-L3
Status Details:      Subscription is current
Subscription Type:   Instance Based
Starts:              02/12/2017
Ends:                02/12/2018
System Type:         Physical


[root@dhcp35-231 facts]# subscription-manager repos --list
This system has no repositories available through subscriptions.

4) NOTE :If the "auto-attach" is enabled on the client (says its set 2min) i see a new active "golden ticket" is getting attached . 

after 2min the consumed list :
=============================

[root@dhcp35-231 ~]# subscription-manager list --consumed
+-------------------------------------------+
   Consumed Subscriptions
+-------------------------------------------+
Subscription Name:   Content Access
Provides:             Content Access
SKU:                 content_access
Contract:            
Account:             
Serial:              842082998312036743 ==> Notice its same serial number
Pool ID:             Not Available
Provides Management: No
Active:              True
Quantity Used:       1
Service Level:       
Service Type:        
Status Details:      Subscription is current
Subscription Type:   
Starts:              02/17/2017
Ends:                02/17/2018
System Type:         Physical

Subscription Name:   Awesome OS Instance Based (Standard Support)
Provides:            Awesome OS Instance Server Bits
SKU:                 awesomeos-instancebased
Contract:            1
Account:             12331131231
Serial:              7010439351232508225
Pool ID:             8ac6a3635a3749c0015a374ad37e0af9
Provides Management: No
Active:              True
Quantity Used:       2
Service Level:       Standard
Service Type:        L1-L3
Status Details:      Subscription is current
Subscription Type:   Instance Based
Starts:              02/12/2017
Ends:                02/12/2018
System Type:         Physical



Actual results:
golden ticket entitlement cert is not immediately active. Also observed that no clock skew were displayed in rhsm.log

Expected results:
Golden ticket entitlement cert should be active

Additional info:
rhsm.logs when auto-attach enabled on the  :
==============================================================

2017-02-17 07:53:26,763 [INFO] rhsmcertd-worker:5774:MainThread @connection.py:755 - Connection built: http_proxy=auto-services.usersys.redhat.com:3128 host=F21-candlepin.usersys.redhat.com port=8443 handler=/candlepin auth=identity_cert ca_dir=/etc/rhsm/ca/ insecure=False
2017-02-17 07:53:29,505 [INFO] rhsmcertd-worker:5774:MainThread @connection.py:505 - Response: status=200, requestUuid=28ba22e4-035f-4f9b-91b7-5465c2abcb76, request="GET /candlepin/consumers/d4a4ed54-6a69-457d-8b29-43c35ded808d"
2017-02-17 07:53:32,242 [INFO] rhsmcertd-worker:5774:MainThread @connection.py:505 - Response: status=200, requestUuid=7a154616-4390-441d-8e86-3b99b7d7e4aa, request="GET /candlepin/consumers/d4a4ed54-6a69-457d-8b29-43c35ded808d/compliance"
2017-02-17 07:53:32,254 [INFO] rhsmcertd-worker:5774:MainThread @cert_sorter.py:205 - Product status: valid_products= partial_products= expired_products= unentitled_producs=32060 future_products= valid_until=None
2017-02-17 07:53:32,256 [WARNING] rhsmcertd-worker:5774:MainThread @healinglib.py:97 - Found invalid entitlements for today: 2017-02-17 12:53:29.509063+00:00
2017-02-17 07:53:36,543 [INFO] rhsmcertd-worker:5774:MainThread @connection.py:505 - Response: status=200, requestUuid=b30322fb-6a22-4cab-96fe-ec490716521f, request="POST /candlepin/consumers/d4a4ed54-6a69-457d-8b29-43c35ded808d/entitlements?entitle_date=2017-02-17T12%3A53%3A29.509063%2B00%3A00"
2017-02-17 07:53:36,549 [INFO] rhsmcertd-worker:5774:MainThread @connection.py:755 - Connection built: http_proxy=auto-services.usersys.redhat.com:3128 host=F21-candlepin.usersys.redhat.com port=8443 handler=/candlepin auth=identity_cert ca_dir=/etc/rhsm/ca/ insecure=False
2017-02-17 07:53:39,333 [INFO] rhsmcertd-worker:5774:MainThread @connection.py:505 - Response: status=200, requestUuid=de760429-d71e-4a75-9545-a8f0a376b56e, request="GET /candlepin/consumers/d4a4ed54-6a69-457d-8b29-43c35ded808d/certificates/serials"
2017-02-17 07:53:42,103 [INFO] rhsmcertd-worker:5774:MainThread @connection.py:505 - Response: status=200, requestUuid=1f5b61ab-a2be-4769-9ceb-848b301b24e1, request="GET /candlepin/consumers/d4a4ed54-6a69-457d-8b29-43c35ded808d/certificates?serials=3791512452974560779"
2017-02-17 07:53:42,115 [INFO] rhsmcertd-worker:5774:MainThread @entcertlib.py:130 - certs updated:
Total updates: 2
Found (local) serial# [2674265043909599500L]
Expected (UEP) serial# [3791512452974560779, 2674265043909599500]
Added (new)
  [sn:3791512452974560779 (Awesome OS Instance Server Bits,) @ /etc/pki/entitlement/3791512452974560779.pem]
  [sn:2674265043909599500 ( Content Access,) @ /etc/pki/entitlement/2674265043909599500.pem]
Deleted (rogue):
  <NONE>
2017-02-17 07:53:45,072 [INFO] rhsmcertd-worker:5774:MainThread @connection.py:505 - Response: status=200, requestUuid=d4898219-9c1a-4954-9b16-a765fe6a10e5, request="GET /candlepin/status"
2017-02-17 07:53:45,166 [INFO] subscription-manager-gui:5732:CertMonitorThread @connection.py:505 - Response: status=200, requestUuid=5f3d9bde-e9e8-4186-9e3b-9eaed26d647a, request="GET /candlepin/consumers/d4a4ed54-6a69-457d-8b29-43c35ded808d/compliance"
2017-02-17 07:53:45,183 [INFO] subscription-manager-gui:5732:CertMonitorThread @cert_sorter.py:205 - Product status: valid_products=32060 partial_products= expired_products= unentitled_producs= future_products= valid_until=2018-02-13 00:00:01+00:00
2017-02-17 07:53:47,818 [INFO] rhsmcertd-worker:5774:MainThread @connection.py:505 - Response: status=304, requestUuid=04b94f91-d890-4441-8ed9-717b62ef9462, request="GET /candlepin/consumers/d4a4ed54-6a69-457d-8b29-43c35ded808d/accessible_content"
2017-02-17 07:53:47,991 [INFO] subscription-manager-gui:5732:CertMonitorThread @connection.py:505 - Response: status=200, requestUuid=cd8ed4f3-92c1-4cc7-9daa-985f2b42f194, request="GET /candlepin/consumers/d4a4ed54-6a69-457d-8b29-43c35ded808d"
2017-02-17 07:53:50,469 [INFO] rhsmcertd-worker:5774:MainThread @connection.py:505 - Response: status=200, requestUuid=c0d1e915-1958-4705-9128-0dedd2c80d0a, request="GET /candlepin/"
2017-02-17 07:53:50,764 [INFO] subscription-manager-gui:5732:CertMonitorThread @connection.py:505 - Response: status=200, requestUuid=187f37fc-ec77-44d2-9717-093dae0a60e8, request="GET /candlepin/consumers/d4a4ed54-6a69-457d-8b29-43c35ded808d"
2017-02-17 07:53:53,262 [INFO] rhsmcertd-worker:5774:MainThread @connection.py:505 - Response: status=200, requestUuid=a51cc63e-7dce-44e3-8f5f-64fb331b441e, request="GET /candlepin/consumers/d4a4ed54-6a69-457d-8b29-43c35ded808d/content_overrides"
2017-02-17 07:53:53,555 [INFO] subscription-manager-gui:5732:CertMonitorThread @connection.py:505 - Response: status=200, requestUuid=0a2406a9-2e1a-47dc-848c-fd7895716e00, request="GET /candlepin/consumers/d4a4ed54-6a69-457d-8b29-43c35ded808d/entitlements?exclude=certificates.key&exclude=certificates.cert"
2017-02-17 07:53:53,620 [INFO] rhsmd:5746:MainThread @connection.py:755 - Connection built: http_proxy=auto-services.usersys.redhat.com:3128 host=F21-candlepin.usersys.redhat.com port=8443 handler=/candlepin auth=identity_cert ca_dir=/etc/rhsm/ca/ insecure=False
2017-02-17 07:53:56,008 [INFO] rhsmcertd-worker:5774:MainThread @connection.py:505 - Response: status=200, requestUuid=9115b8d7-70e4-4c37-b6a6-fccdb3dd8e85, request="GET /candlepin/consumers/d4a4ed54-6a69-457d-8b29-43c35ded808d/release"
2017-02-17 07:53:56,028 [INFO] rhsmcertd-worker:5774:MainThread @repolib.py:310 - repos updated: Repo updates

Total repo updates: 23
Updated
    <NONE>
Added (new)
    [id:awesomeos-ia64-only-content awesomeos-ia64-only-content]
    [id:awesomeos-i386-only-content awesomeos-i386-only-content]
    [id:snowy-content-label snowy-content]
    [id:never-enabled-content never-enabled-content]
    [id:awesomeos-all awesomeos-all]
    [id:awesomeos-ppc-only-content awesomeos-ppc-only-content]
    [id:awesomeos-x86_64-only-content awesomeos-x86_64-only-content]
    [id:awesomeos-ppc64 awesomeos-ppc64]
    [id:awesomeos-s390x awesomeos-s390x]
    [id:awesomeos-ia64 awesomeos-ia64]
    [id:awesomeos-i686 awesomeos-i686]
    [id:awesomeos-x86_64 awesomeos-x86_64]
    [id:awesomeos-ppc64-only-content awesomeos-ppc64-only-content]
    [id:awesomeos-s390x-only-content awesomeos-s390x-only-content]
    [id:awesomeos-x86_64-i386-content awesomeos-x86_64-i386-content]
    [id:content-label-empty-gpg content-emptygpg]
    [id:awesomeos-i386 awesomeos-i386]
    [id:awesomeos-x86 awesomeos-x86]
    [id:awesomeos awesomeos]
    [id:awesomeos-ppc awesomeos-ppc]
    [id:content-label content]
    [id:snowy-never-enabled-content snowy-never-enabled-content]
    [id:content-label-no-gpg content-nogpg]
Deleted
    <NONE>
2017-02-17 07:53:56,029 [INFO] rhsmcertd-worker:5774:MainThread @container.py:164 - Copying: /etc/pki/entitlement/2674265043909599500.pem -> /etc/docker/certs.d/registry.access.redhat.com/2674265043909599500.cert
2017-02-17 07:53:56,029 [INFO] rhsmcertd-worker:5774:MainThread @container.py:170 - Copying: /etc/pki/entitlement/2674265043909599500-key.pem -> /etc/docker/certs.d/registry.access.redhat.com/2674265043909599500.key
2017-02-17 07:53:56,029 [INFO] rhsmcertd-worker:5774:MainThread @container.py:164 - Copying: /etc/pki/entitlement/2674265043909599500.pem -> /etc/docker/certs.d/cdn.redhat.com/2674265043909599500.cert
2017-02-17 07:53:56,030 [INFO] rhsmcertd-worker:5774:MainThread @container.py:170 - Copying: /etc/pki/entitlement/2674265043909599500-key.pem -> /etc/docker/certs.d/cdn.redhat.com/2674265043909599500.key
2017-02-17 07:53:56,030 [INFO] rhsmcertd-worker:5774:MainThread @container.py:164 - Copying: /etc/pki/entitlement/2674265043909599500.pem -> /etc/docker/certs.d/access.redhat.com/2674265043909599500.cert
2017-02-17 07:53:56,030 [INFO] rhsmcertd-worker:5774:MainThread @container.py:170 - Copying: /etc/pki/entitlement/2674265043909599500-key.pem -> /etc/docker/certs.d/access.redhat.com/2674265043909599500.key
2017-02-17 07:53:56,031 [INFO] rhsmcertd-worker:5774:MainThread @container.py:164 - Copying: /etc/pki/entitlement/2674265043909599500.pem -> /etc/docker/certs.d/registry.redhat.io/2674265043909599500.cert
2017-02-17 07:53:56,031 [INFO] rhsmcertd-worker:5774:MainThread @container.py:170 - Copying: /etc/pki/entitlement/2674265043909599500-key.pem -> /etc/docker/certs.d/registry.redhat.io/2674265043909599500.key
2017-02-17 07:53:56,039 [INFO] rhsmcertd-worker:5774:MainThread @healinglib.py:131 - Entitlement auto healing was checked and entitlements
2017-02-17 07:53:56,363 [INFO] rhsmd:5746:MainThread @connection.py:505 - Response: status=200, requestUuid=12180b69-e399-445b-b990-91c123c926c0, request="GET /candlepin/consumers/d4a4ed54-6a69-457d-8b29-43c35ded808d/compliance"
2017-02-17 07:53:56,394 [INFO] rhsmd:5746:MainThread @cert_sorter.py:205 - Product status: valid_products=32060 partial_products= expired_products= unentitled_producs= future_products= valid_until=2018-02-13 00:00:01+00:00
2017-02-17 07:53:58,820 [INFO] rhsmcertd-worker:5774:MainThread @connection.py:505 - Response: status=200, requestUuid=5a3e236f-b659-4afb-ac4c-53121ef92a6e, request="GET /candlepin/consumers/d4a4ed54-6a69-457d-8b29-43c35ded808d/certificates/serials"
2017-02-17 07:53:58,820 [INFO] rhsmcertd-worker:5774:MainThread @entcertlib.py:130 - certs updated:
Total updates: 0
Found (local) serial# [2674265043909599500L, 3791512452974560779L]
Expected (UEP) serial# [3791512452974560779, 2674265043909599500]
Added (new)
  <NONE>
Deleted (rogue):
  <NONE>
2017-02-17 07:54:01,549 [INFO] rhsmcertd-worker:5774:MainThread @connection.py:505 - Response: status=304, requestUuid=87093e19-11b1-437b-9476-fa65435e0a36, request="GET /candlepin/consumers/d4a4ed54-6a69-457d-8b29-43c35ded808d/accessible_content"
2017-02-17 07:54:28,389 [INFO] rhsmd:5785:MainThread @rhsmd:263 - rhsmd started
2017-02-17 07:54:28,483 [INFO] subscription-manager:5783:MainThread @managercli.py:390 - Client Versions: {'python-rhsm': '1.19.1-1.git.3.9985682.el6', 'subscription-manager': '1.19.1-1.git.9.908316a.el6'}
2017-02-17 07:54:28,484 [INFO] subscription-manager:5783:MainThread @connection.py:755 - Connection built: http_proxy=auto-services.usersys.redhat.com:3128 host=F21-candlepin.usersys.redhat.com port=8443 handler=/candlepin auth=identity_cert ca_dir=/etc/rhsm/ca/ insecure=False
2017-02-17 07:54:28,484 [INFO] subscription-manager:5783:MainThread @connection.py:755 - Connection built: http_proxy=auto-services.usersys.redhat.com:3128 host=F21-candlepin.usersys.redhat.com port=8443 handler=/candlepin auth=none



rhsm.logs when disabled the auto-attach process:=
==========================================

2017-02-17 07:48:26,661 [INFO] rhsmcertd-worker:5755:MainThread @connection.py:755 - Connection built: http_proxy=auto-services.usersys.redhat.com:3128 host=F21-candlepin.usersys.redhat.com port=8443 handler=/candlepin auth=identity_cert ca_dir=/etc/rhsm/ca/ insecure=False
2017-02-17 07:48:29,710 [INFO] rhsmcertd-worker:5755:MainThread @connection.py:505 - Response: status=200, requestUuid=ce874117-130a-4af8-b493-7018da73af11, request="GET /candlepin/consumers/d4a4ed54-6a69-457d-8b29-43c35ded808d"
2017-02-17 07:48:29,713 [INFO] rhsmcertd-worker:5755:MainThread @healinglib.py:78 - Auto-heal disabled on server, skipping.
2017-02-17 07:48:32,536 [INFO] rhsmcertd-worker:5755:MainThread @connection.py:505 - Response: status=200, requestUuid=ab0c8dc1-d1f8-4919-b76c-d01d130d5b97, request="GET /candlepin/consumers/d4a4ed54-6a69-457d-8b29-43c35ded808d/certificates/serials"
2017-02-17 07:48:32,537 [INFO] rhsmcertd-worker:5755:MainThread @entcertlib.py:130 - certs updated:
Total updates: 0
Found (local) serial# [2674265043909599500L]
Expected (UEP) serial# [2674265043909599500]
Added (new)
  <NONE>
Deleted (rogue):
  <NONE>
2017-02-17 07:48:35,205 [INFO] rhsmcertd-worker:5755:MainThread @connection.py:505 - Response: status=200, requestUuid=a245843a-7d1d-4f46-a9e2-26a46c34409f, request="GET /candlepin/status"
2017-02-17 07:48:38,045 [INFO] rhsmcertd-worker:5755:MainThread @connection.py:505 - Response: status=304, requestUuid=35d3bb06-fba2-4bd3-83da-dc0310329be3, request="GET /candlepin/consumers/d4a4ed54-6a69-457d-8b29-43c35ded808d/accessible_content"
2017-02-17 07:48:58,679 [INFO] subscription-manager:5756:MainThread @managercli.py:390 - Client Versions: {'python-rhsm': '1.19.1-1.git.3.9985682.el6', 'subscription-manager': '1.19.1-1.git.9.908316a.el6'}
2017-02-17 07:48:58,680 [INFO] subscription-manager:5756:MainThread @connection.py:755 - Connection built: http_proxy=auto-services.usersys.redhat.com:3128 host=F21-candlepin.usersys.redhat.com port=8443 handler=/candlepin auth=identity_cert ca_dir=/etc/rhsm/ca/ insecure=False
2017-02-17 07:48:58,681 [INFO] subscription-manager:5756:MainThread @connection.py:755 - Connection built: http_proxy=auto-services.usersys.redhat.com:3128 host=F21-candlepin.usersys.redhat.com port=8443 handler=/candlepin auth=none
2017-02-17 07:49:01,412 [INFO] subscription-manager:5756:MainThread @connection.py:505 - Response: status=200, requestUuid=c435292a-b141-4115-8c9e-4e45e69433c9, request="GET /candlepin/consumers/d4a4ed54-6a69-457d-8b29-43c35ded808d/compliance"
2017-02-17 07:49:01,413 [INFO] subscription-manager:5756:MainThread @cert_sorter.py:205 - Product status: valid_products= partial_products= expired_products= unentitled_producs=32060 future_products= valid_until=None