Red Hat Satellite engineering is moving the tracking of its product development work on Satellite to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "Satellite project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs will be migrated starting at the end of May. If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "Satellite project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/SAT-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1631590 - Candlepin throws 500 Internal Server Error for more than 40+ guests
Summary: Candlepin throws 500 Internal Server Error for more than 40+ guests
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Satellite
Classification: Red Hat
Component: Candlepin
Version: 6.4
Hardware: Unspecified
OS: Unspecified
unspecified
urgent
Target Milestone: 6.4.0
Assignee: Justin Sherrill
QA Contact: Jitendra Yejare
URL:
Whiteboard:
Depends On: 1632764
Blocks: 1586210
TreeView+ depends on / blocked
 
Reported: 2018-09-21 02:05 UTC by Eko
Modified: 2019-11-05 23:35 UTC (History)
12 users (show)

Fixed In Version: candlepin-2.4.8-1,tfm-rubygem-katello-3.7.0.41-1
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1632761 1632764 1633252 (view as bug list)
Environment:
Last Closed: 2018-10-16 19:16:43 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Foreman Issue Tracker 25026 0 Normal Closed Candlepin throws 500 Internal Server Error for more than 40+ guests 2020-05-15 12:05:11 UTC
Foreman Issue Tracker 25060 0 Normal Closed do not send large bulk requests for consumers as part of virt-who check in 2020-05-15 12:05:12 UTC
Github candlepin candlepin pull 2120 0 'None' closed [M] 1631590: Added limits to the size of link headers (ENT-893) 2020-05-15 12:05:12 UTC

Comment 4 Jitendra Yejare 2018-09-21 08:00:39 UTC 
Hi Eko,


The Package Versions:

virt-what-1.18-4.el7.x86_64
tfm-rubygem-hammer_cli_foreman_virt_who_configure-0.0.3-2.el7sat.noarch
tfm-rubygem-foreman_virt_who_configure-0.2.2-1.el7sat.noarch



Also, I don't see any logs under satellites /var/log/rhsm/rhsm.log during posting.



Note:
------
I can easily post upto 30 virt guests, but the issue comes when it is more than or equal to 40.
Comment 5 Eko 2018-09-21 08:18:08 UTC 
Hi Jitendra,
it's not virt-what package, it should be virt-who, please check again.

and /var/log/rhsm/rhsm.log should be in the host which virt-who was installed.
Comment 6 Eko 2018-09-21 08:55:28 UTC 
After discussing with Jitendra, virt-who package never be installed and used for this issue, so I'm afraid it's not a virt-who bug.

According to the error log message, I will move it to the candlepin component to check again.
Comment 8 Mike McCune 2018-09-21 20:58:34 UTC 
This does appear to be natively in Candlepin, after posting directly to the virt-who API endpoint the error occurs easily. 

We can tune around this via configs in server.xml, I had to get up into the 100MB range before 100 hosts would work:

               maxHttpHeaderSize="100000"

that is quite a bit more than the default of 4MB which indicates to me that something isn't really working correctly in formulating the response back to the API call.

Customers with 1000 virtual machines in a virt-who transaction would overwhelm the server and require 1G+ ram in the response which .. is a bit much for a HTTP response.

We need to examine how we are formulating the response headers in this API call.
Comment 9 Mike McCune 2018-09-21 21:04:17 UTC 
Reproducer info:

I used https://hub.docker.com/r/jacobcallahan/genvirt/ to generate the API call.
Comment 13 Justin Sherrill 2018-09-24 16:33:29 UTC 
Created redmine issue https://projects.theforeman.org/issues/25026 from this bug
Comment 14 Brad Buckingham 2018-09-24 20:33:47 UTC 
The upstream katello PR has now merged as well.  Moving the BZ to POST.
Comment 17 Jitendra Yejare 2018-09-27 14:13:07 UTC 
FailedQA!

Steps:
--------

1. Post 1000/300/200 hypervisors json to rhsm/hypervisors using https://hub.docker.com/r/jacobcallahan/genvirt


Observation:
--------------

500 ISE Error : https://pastebin.com/YrpBeArD


Note:
------------

The post of 100 hypervisors json works.


Already discussed and shared details with jsherrill, crog.
Comment 20 Brad Buckingham 2018-09-28 12:08:12 UTC 
PR merged upstream; therefore, moving BZ to POST.
Comment 21 Jitendra Yejare 2018-09-28 20:50:02 UTC 
Tested the patch downstrean on RC1(snap 24), It works nicely.

Tried posting 200 and max 1000 at a time, nothing breaks works perfect.
Comment 23 Mike McCune 2018-10-02 16:25:04 UTC 
tested this with 5k hosts without error:

#   docker run -e "SATHOST=sat-r220-01.lab.eng.rdu2.redhat.com" -e "COUNT=5000" jacobcallahan/genvirt
...
Generating data with 5000 hosts.
Submitting data to sat-r220-01.lab.eng.rdu2.redhat.com. This may take a while...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  771k    0     4  100  771k      0    494  0:26:39  0:26:38  0:00:01     0
nullUnregistering from Satellite
Unregistering from: sat-r220-01.lab.eng.rdu2.redhat.com:443/rhsm
System has been unregistered.
Done!
Comment 25 Bryan Kearney 2018-10-16 19:16:43 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2018:2927
Note You need to log in before you can comment on or make changes to this bug.