Bug 1564452 - Registering of Hypervisors on large scale with virt-who fails with candlepin closing the PgSQL connections unexpectedly
Summary: Registering of Hypervisors on large scale with virt-who fails with candlepin ...
Keywords:
Status: CLOSED WONTFIX
Alias: None
Product: Red Hat Satellite
Classification: Red Hat
Component: Candlepin
Version: 6.3.0
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: Unspecified
Assignee: satellite6-bugs
QA Contact: jcallaha
URL:
Whiteboard: scale_lab
Depends On: 1564568 1694809
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-04-06 10:33 UTC by sbadhwar
Modified: 2019-06-10 14:38 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1564568 (view as bug list)
Environment:
Last Closed: 2019-06-03 12:05:28 UTC
Target Upstream Version:


Attachments (Terms of Use)
Candlepin error log (408.59 KB, text/plain)
2018-04-06 10:33 UTC, sbadhwar
no flags Details
Postgres log (31.91 KB, text/plain)
2018-04-06 10:34 UTC, sbadhwar
no flags Details

Description sbadhwar 2018-04-06 10:33:47 UTC
Created attachment 1418054 [details]
Candlepin error log

Description of problem:
While trying to report hypervisor data using virt-who to satellite, the reporting suffered an error.

The following data was being reported: 300 Hosts and 19K VMs running over VMWare ESXi.

Virt-who reported the following in debug mode:
2018-04-06 06:11:17,790 [rhsm.connection DEBUG] MainProcess(1860):Thread-4 @connection.py:_request:516 - Making request: POST /rhsm/hypervisors/Default_Organization?reporter_id=gprfc023.sbu.lab.eng.bos.redhat.com-e542c5cdcd5a42b9afddad8b4529e928&cloaked=False&env=Library
2018-04-06 06:11:29,395 [rhsm.connection INFO] MainProcess(1860):Thread-4 @connection.py:_request:552 - Response: status=200, request="POST /rhsm/hypervisors/Default_Organization?reporter_id=gprfc023.sbu.lab.eng.bos.redhat.com-e542c5cdcd5a42b9afddad8b4529e928&cloaked=False&env=Library"
2018-04-06 06:11:29,421 [virtwho.destination_8759358284855132892 DEBUG] MainProcess(1860):Thread-4 @subscriptionmanager.py:_connect:128 - Authenticating with RHSM username admin
2018-04-06 06:11:29,424 [rhsm.connection INFO] MainProcess(1860):Thread-4 @connection.py:__init__:822 - Connection built: host=gprfc018.sbu.lab.eng.bos.redhat.com port=443 handler=/rhsm auth=basic username=admin
2018-04-06 06:11:29,426 [rhsm.connection DEBUG] MainProcess(1860):Thread-4 @connection.py:_request:516 - Making request: GET /rhsm/status/
2018-04-06 06:11:29,473 [rhsm.connection INFO] MainProcess(1860):Thread-4 @connection.py:_request:552 - Response: status=200, request="GET /rhsm/status/"
2018-04-06 06:11:29,474 [virtwho.destination_8759358284855132892 DEBUG] MainProcess(1860):Thread-4 @subscriptionmanager.py:check_report_state:263 - Checking status of job hypervisor_update_e722e400-a203-495b-a9c5-9c423e8b0ed1
2018-04-06 06:11:29,474 [rhsm.connection DEBUG] MainProcess(1860):Thread-4 @connection.py:_request:516 - Making request: GET /rhsm/jobs/hypervisor_update_e722e400-a203-495b-a9c5-9c423e8b0ed1?result_data=True
2018-04-06 06:11:29,533 [rhsm.connection INFO] MainProcess(1860):Thread-4 @connection.py:_request:552 - Response: status=200, request="GET /rhsm/jobs/hypervisor_update_e722e400-a203-495b-a9c5-9c423e8b0ed1?result_data=True"
2018-04-06 06:11:29,533 [virtwho.destination_8759358284855132892 WARNING] MainProcess(1860):Thread-4 @subscriptionmanager.py:check_report_state:282 - Job status report without resultData: {u'finishTime': None, u'targetType': u'owner', u'updated': u'2018-04-06T10:11:29+0000', u'group': u'async group', u'statusPath': u'/jobs/hypervisor_update_e722e400-a203-495b-a9c5-9c423e8b0ed1', u'resultData': None, u'created': u'2018-04-06T10:11:29+0000', u'targetId': u'Default_Organization', u'principalName': u'foreman_admin', u'state': u'FAILED', u'done': True, u'result': u'javax.persistence.RollbackException: Error while committing the transaction', u'startTime': u'2018-04-06T10:11:29+0000', u'ownerId': u'Default_Organization', u'id': u'hypervisor_update_e722e400-a203-495b-a9c5-9c423e8b0ed1', u'correlationId': u'feb97bf4'}
2018-04-06 06:11:29,545 [virtwho.destination_8759358284855132892 DEBUG] MainProcess(1860):Thread-4 @virt.py:run:394 - Thread 'destination_8759358284855132892' terminated
2018-04-06 06:11:29,803 [virtwho.main DEBUG] MainProcess(1860):MainThread @__main__.py:main:23 - virt-who terminated
2018-04-06 06:11:29,803 [virtwho.main DEBUG] MainProcess(1860):MainThread @executor.py:terminate:226 - virt-who is shutting down


On checking the task status, the following data was obtained from the API endpoint:
{
  "id": "hypervisor_update_e722e400-a203-495b-a9c5-9c423e8b0ed1",
  "state": "FAILED",
  "startTime": "2018-04-06T10:11:29+0000",
  "finishTime": "2018-04-06T10:11:29+0000",
  "result": "javax.persistence.RollbackException: Error while committing the transaction",
  "principalName": "foreman_admin",
  "targetType": "owner",
  "targetId": "Default_Organization",
  "ownerId": "Default_Organization",
  "correlationId": "feb97bf4",
  "resultData": "[cloaked]",
  "statusPath": "/jobs/hypervisor_update_e722e400-a203-495b-a9c5-9c423e8b0ed1",
  "done": true,
  "group": "async group",
  "created": "2018-04-06T10:11:29+0000",
  "updated": "2018-04-06T10:11:32+0000"
}



Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:
The virt-who reporting ends with an error in the reporting task

Expected results:
virt-who reporting ends successfully

Additional info:

Comment 1 sbadhwar 2018-04-06 10:34:19 UTC
Created attachment 1418055 [details]
Postgres log

Comment 2 Barnaby Court 2018-04-06 15:52:49 UTC
Not sure that this is actually scale related so much as sample data related. I'm cloning upstream given the strange exception in the candlepin log file.

Comment 4 Bryan Kearney 2019-05-02 19:06:47 UTC
The Satellite Team is attempting to provide an accurate backlog of bugzilla requests which we feel will be resolved in the next few releases. We do not believe this bugzilla will meet that criteria, and have plans to close it out in 1 month. This is not a reflection on the validity of the request, but a reflection of the many priorities for the product. If you have any concerns about this, feel free to contact Red Hat Technical Support or your account team. If we do not hear from you, we will close this bug out. Thank you.

Comment 5 Bryan Kearney 2019-06-03 12:05:28 UTC
Thank you for your interest in Satellite 6. We have evaluated this request, and while we recognize that it is a valid request, we do not expect this to be implemented in the product in the foreseeable future. This is due to other priorities for the product, and not a reflection on the request itself. We are therefore closing this out as WONTFIX. If you have any concerns about this, please do not reopen. Instead, feel free to contact Red Hat Technical Support. Thank you.


Note You need to log in before you can comment on or make changes to this bug.