Red Hat Satellite engineering is moving the tracking of its product development work on Satellite to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "Satellite project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs will be migrated starting at the end of May. If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "Satellite project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/SAT-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1369082 - Large virt-who json may cause performance issue
Summary: Large virt-who json may cause performance issue
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Satellite
Classification: Red Hat
Component: Content Management
Version: 6.2.0
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: Unspecified
Assignee: Justin Sherrill
QA Contact: Perry Gagne
URL: http://projects.theforeman.org/issues...
Whiteboard:
Depends On:
Blocks: 1353215 1385860 1389103 1389104
TreeView+ depends on / blocked
 
Reported: 2016-08-22 12:55 UTC by Justin Sherrill
Modified: 2020-04-15 14:37 UTC (History)
19 users (show)

Fixed In Version: tfm-rubygem-katello-3.0.0.82-1
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1385860 1389103 1389104 (view as bug list)
Environment:
Last Closed: 2016-11-10 08:23:06 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)
tfm-rubygem-katello-3.0.0.80-2.bz1369082_1357878.el7sat.noarch.rpm (4.84 MB, application/x-rpm)
2016-10-18 15:48 UTC, Zach Huntington-Meath 		no flags Details
tfm-rubygem-katello-3.0.0.80-2.1369082_1357878.el6sat.noarch.rpm (5.44 MB, application/x-rpm)
2016-10-21 17:06 UTC, Zach Huntington-Meath 		no flags Details
tfm-rubygem-katello-3.0.0.81-2.BZ_1368746_1365952.el7sat.noarch.rpm (4.85 MB, application/x-rpm)
2016-11-04 15:24 UTC, Zach Huntington-Meath 		no flags Details
tfm-rubygem-katello_ostree-3.0.0.81-2.BZ_1368746_1365952.el7sat.noarch.rpm (102.38 KB, application/x-rpm)
2016-11-04 15:25 UTC, Zach Huntington-Meath 		no flags Details
View All


Links
System ID Private Priority Status Summary Last Updated
Foreman Issue Tracker 16228 0 Normal Closed Large virt-who json may cause performance issue 2020-09-02 15:21:00 UTC
Red Hat Knowledge Base (Solution) 2755731 0 None None None 2016-11-07 17:29:36 UTC
Red Hat Product Errata RHBA-2016:2700 0 normal SHIPPED_LIVE Satellite 6.2.4 Tools Async Release 2016-11-10 13:22:25 UTC

Description Justin Sherrill 2016-08-22 12:55:28 UTC 
Description of problem:

A Very large hypervisor json checkin from virt-who may cause a performance issue.  We should not save the full json from a virt-who checkin as task input


Version-Release number of selected component (if applicable):


How reproducible:
Always

Steps to Reproduce:
1.  Have virt-who checkin
2.  Go to monitor > tasks, click on the hypervisors update task
3.  Click the dynflow button
4.  Click the finalize tab, click the action step

Actual results:
see full virt-who json

Expected results:
no full virt-who json.  Just save the pieces of info we need

Additional info:
Comment 1 Justin Sherrill 2016-08-22 13:55:10 UTC 
Created redmine issue http://projects.theforeman.org/issues/16228 from this bug
Comment 2 Bryan Kearney 2016-08-22 16:18:32 UTC 
Upstream bug assigned to jsherril
Comment 3 Bryan Kearney 2016-08-22 16:18:34 UTC 
Upstream bug component is Content Management
Comment 4 Bryan Kearney 2016-08-22 16:18:36 UTC 
Upstream bug assigned to jsherril
Comment 6 Bryan Kearney 2016-09-13 22:17:41 UTC 
Moving this bug to POST for triage into Satellite 6 since the upstream issue http://projects.theforeman.org/issues/16228 has been resolved.
Comment 11 Eko 2016-10-17 09:31:30 UTC 
Could you help to provide more details?

1. virt-who package version?

2. virt-who configure options

==== for virt-who-0.17 ====
1). if the VIRTWHO_DEBUG option is disabled, the json info should be not printed out 

2). about the checkin interval time, if VIRTWHO_INTERVAL <=0,  the default checkin time should be 60s, if VIRTWHO_INTERVAL > 60, will checkin as the setting value.
Comment 12 Mike McCune 2016-10-17 19:57:36 UTC 
We are seeing cases where the Hypervisor task in Katello consumes *huge* amount of space in the dynflow_actions table because we probably store all the JSON from the virt-who transaction for every single API call.
Comment 13 Zach Huntington-Meath 2016-10-18 15:47:18 UTC 
HOTFIX INSTRUCTIONS

I've produced hotfix packages for the customer's installed version
of Satellite (6.2.2.1 on el7) and tested on a similar Satellite. With this you will not receive the whole JSON output from virt-who in the task.

Install instructions:


1) Download the rpm attached to this Bugzilla to your satellite server.

2) Stop katello-service

katello-service stopls


3) Install packages

yum localinstall tfm-rubygem-katello-3.0.0.80-2.bz1369082_1357878.el7sat.noarch.rpm

4) Start katello-service

katello-service- start

This should properly start Katello. Now if you check check the hypervisors task now it should only have a small amount of data now recognizing that virt-who has checked in.
Comment 14 Zach Huntington-Meath 2016-10-18 15:48:11 UTC 
Created attachment 1211777 [details]
tfm-rubygem-katello-3.0.0.80-2.bz1369082_1357878.el7sat.noarch.rpm
Comment 16 Mike McCune 2016-10-19 18:30:13 UTC 
> --- Comment #15 from jcallaha ---
> It looks like there are two issues here.
> 
> 1. Large virt-who check-in performance issues.
> 
> 2. Repeated check-ins when actions are performed on the virt backend.

actually just (1) that we are resolving in this bug. The frequency of the actual tasks isn't at issue, it is the size of the task that is stored in the DB.

> 
> If that is correct, then I Have a few questions.
> 
> 1. Can we use a large json input to simulate the large check-in? 
> 1.a.If not, then does anyone have access to a large live system? 
> 1.b. This might be unnecessary anyway as long as we don't see the full json in
> the task, correct?
> 

yup, will get one attached to this BZ that you can use.


> 2. Would the repeated checkin tasks be seen in the foreman-tasks page?
> 2.a. If so, then what exactly should we be looking for (no tasks, minimal
> tasks)?
> 2.b. If not, then where should we be looking to make sure the behavior is what
> we are expecting?
> 

IMO and Justin can also comment:

what we would want todo is check the size of the task object when running an API call on the hypervisor task

Before this fix, customer saw a 10MB+ JSON extract on one task when running:

# curl -k -u admin:changeme "https://localhost/foreman_tasks/api/tasks?page=1&per_page=1"

before this fix it was a very large file, even for one task

after the fix it should be KB, not MB in size.
Comment 17 Zach Huntington-Meath 2016-10-21 17:06:35 UTC 
Created attachment 1212914 [details]
tfm-rubygem-katello-3.0.0.80-2.1369082_1357878.el6sat.noarch.rpm
Comment 19 Zach Huntington-Meath 2016-11-04 15:20:55 UTC 
HOTFIX INSTRUCTIONS

I've produced hotfix packages for the customer's installed version
of Satellite (6.2.3 on el7) and tested on a similar Satellite. With this you will not receive the whole JSON output from virt-who in the task.

Install instructions:


1) Download the following rpms attached to this Bugzilla to your satellite server.
tfm-rubygem-katello-3.0.0.81-2.BZ_1368746_1365952.el7sat.noarch.rpm
tfm-rubygem-katello_ostree-3.0.0.81-2.BZ_1368746_1365952.el7sat.noarch.rpm

2) Stop katello-service

katello-service stopls


3) Install packages

yum localinstall tfm-rubygem-katello-3.0.0.81-2.BZ_1368746_1365952.noarch.rpm tfm-rubygem-katello_ostree-3.0.0.81-2.BZ_1368746_1365952.el7sat.noarch.rpm

4) Start katello-service

katello-service- start

This should properly start Katello. Now if you check check the hypervisors task now it should only have a small amount of data now recognizing that virt-who has checked in.

Remeber this is Satellite 6.2.3, if you need this hotfix for another version please contact us.
Comment 20 Zach Huntington-Meath 2016-11-04 15:24:16 UTC 
Created attachment 1217424 [details]
tfm-rubygem-katello-3.0.0.81-2.BZ_1368746_1365952.el7sat.noarch.rpm
Comment 21 Zach Huntington-Meath 2016-11-04 15:25:22 UTC 
Created attachment 1217426 [details]
tfm-rubygem-katello_ostree-3.0.0.81-2.BZ_1368746_1365952.el7sat.noarch.rpm
Comment 23 Daniele Palumbo 2016-11-07 11:42:48 UTC 
I see that tfm-rubygem-katello_ostree has been added into the list of patched packages and the bugzilla that should be fixed are different into the file name.

Can you please explain the difference between the two patches for 6.2.2.1 and 6.2.3?
Comment 26 errata-xmlrpc 2016-11-10 08:23:06 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2016:2700
Comment 27 Justin Sherrill 2017-02-20 22:37:19 UTC 
The two posted rpms were hotfixes and not the final released rpms.  That is how the normal hotfix process goes. 6.2.3 includes many more fixes not available with just the hotfix, the patch for this particular issue would be the same
Note You need to log in before you can comment on or make changes to this bug.