+++ This bug was initially created as a clone of Bug #1631590 +++

Copy the issue from Jitendra Yejare <jyejare@redhat.com>,

Trying to post 40 virt-who guests to /rhsm/hypervisors but Satellite has thrown ISE Error and the guests were not posted.

Please see production.logs [1]  and candlepin.logs [2] or attachment.

[1] : https://pastebin.com/VjeZgjG5
[2] : https://pastebin.com/ceyTb1i7

--- Additional comment from hsun@redhat.com on 20180921T02:05:42

Created attachment 1485370 [details]
virtwho_client_concurrency_erroed

--- Additional comment from hsun@redhat.com on 20180921T02:13:46

We didn't use a large number of guests for virt-who + satellite6.4 testing before, according to the log message, it seems this issue is caused by the buffer size or maxHttpHeaderSize of candlepin or apache settings. 

2018-09-20 11:03:32,875 [thread=http-bio-8443-exec-10] [req=c8d4b309-a10b-41cf-ada1-24c9d489fe66, org=, csid=] ERROR org.candlepin.common.exceptions.mappers.CandlepinExceptionMapper - Runtime Error An attempt was made to write more data to the response headers than there was room available in the buffer. Increase maxHttpHeaderSize on the connector or write less data into the response headers. at org.apache.coyote.http11.AbstractOutputBuffer.checkLengthBeforeWrite:547
org.apache.coyote.http11.HeadersTooLargeException: An attempt was made to write more data to the response headers than there was room available in the buffer. Increase maxHttpHeaderSize on the connector or write less data into the response headers.


@kevin, could you help to check this? or any suggestion? Thanks.

--- Additional comment from hsun@redhat.com on 20180921T02:17:46

Hi, Jitendra Yejare,

Could you help to provide the below detailed info:

1. virt-who package version

2. /var/log/rhsm.log

--- Additional comment from jyejare@redhat.com on 20180921T08:00:39

Hi Eko,


The Package Versions:

virt-what-1.18-4.el7.x86_64
tfm-rubygem-hammer_cli_foreman_virt_who_configure-0.0.3-2.el7sat.noarch
tfm-rubygem-foreman_virt_who_configure-0.2.2-1.el7sat.noarch



Also, I don't see any logs under satellites /var/log/rhsm/rhsm.log during posting.



Note:
------
I can easily post upto 30 virt guests, but the issue comes when it is more than or equal to 40.

--- Additional comment from hsun@redhat.com on 20180921T08:18:08

Hi Jitendra,
it's not virt-what package, it should be virt-who, please check again.

and /var/log/rhsm/rhsm.log should be in the host which virt-who was installed.

--- Additional comment from hsun@redhat.com on 20180921T08:55:28

After discussing with Jitendra, virt-who package never be installed and used for this issue, so I'm afraid it's not a virt-who bug.

According to the error log message, I will move it to the candlepin component to check again.

--- Additional comment from pm-rhel@redhat.com on 20180921T15:15:41

Since this issue was entered in Red Hat Bugzilla, the pm_ack has been
set to + automatically for the next planned release

--- Additional comment from mmccune@redhat.com on 20180921T20:58:34

This does appear to be natively in Candlepin, after posting directly to the virt-who API endpoint the error occurs easily. 

We can tune around this via configs in server.xml, I had to get up into the 100MB range before 100 hosts would work:

               maxHttpHeaderSize="100000"

that is quite a bit more than the default of 4MB which indicates to me that something isn't really working correctly in formulating the response back to the API call.

Customers with 1000 virtual machines in a virt-who transaction would overwhelm the server and require 1G+ ram in the response which .. is a bit much for a HTTP response.

We need to examine how we are formulating the response headers in this API call.

--- Additional comment from mmccune@redhat.com on 20180921T21:04:17

Reproducer info:

I used https://hub.docker.com/r/jacobcallahan/genvirt/ to generate the API call.

--- Additional comment from crog@redhat.com on 20180921T21:57:34

As an aside to this issue, the request which lead to the error in question will not work as expected.

The intent of the request looks to be to fetch a collection of consumers by UUID. However, UUIDs are provided in a doubly-URI-encoded, JSON-serialized array, rather than the expected format of "?uuid=:uuid1&uuid=:uuid2&...&uuid=:uuidN".

I've posted a patch to address the HeadersTooLargeException, though if the request being issued to Candlepin is not fixed, the core of this bug (not getting consumer details back) will persist.

--- Additional comment from bbuckingham@redhat.com on 20180924T13:08:25

It appears that the following commit may have introduced the behavior observed in the URL:

https://github.com/Katello/katello/commit/e0d2c8790718e5aaf366a76b33ee446c06999bde#diff-bf897becee6d218f2e9b589c5f66dcfdR37

--- Additional comment from bbuckingham@redhat.com on 20180924T13:56:18

Based on comment 10 and irc discussion, we are going to re-use this same BZ to include a fix for the katello part.  To support that, I am going to move the bugzilla back to ASSIGNED and over to Justin, as he is investigating on katello.

--- Additional comment from jsherril@redhat.com on 20180924T16:33:29

Created redmine issue https://projects.theforeman.org/issues/25026 from this bug

--- Additional comment from bbuckingham@redhat.com on 20180924T20:33:47

The upstream katello PR has now merged as well.  Moving the BZ to POST.

--- Additional comment from sat6-jenkins@redhat.com on 20180925T18:56:31

build status: succeeded

brew:
 * rubygem-katello: closed - https://brewweb.engineering.redhat.com/brew/taskinfo?taskID=18494435