Bug 1524417

Summary: "apipie:cache:index" task is getting executed two times while updating packages from 6.1 to 6.2 using yum update.
Product: Red Hat Satellite Reporter: sandeep mutkule <smutkule>
Component: UpgradesAssignee: satellite6-bugs <satellite6-bugs>
Status: CLOSED WONTFIX QA Contact: Nikhil Kathole <nkathole>
Severity: medium Docs Contact:
Priority: medium    
Version: 6.2.12CC: ahumbe, apatel, inecas, kgaikwad, mbacovsk, rakumar, smutkule
Target Milestone: UnspecifiedKeywords: Triaged
Target Release: Unused   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-02-28 19:35:16 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
Image attached none

Description sandeep mutkule 2017-12-11 12:56:19 UTC
Description of problem:

While upgrading satellite to its latest version  "apipie:cache:index" process runs twice. Consumes too much time which should be reduced.  


Version-Release number of selected component (if applicable): satellite 6.2


How reproducible: 
1)Run below command to upgrade satellite:

satellite-installer  --scenario  satellite --upgrade 


Actual results: 
Takes too much time to upgrade 


Expected results: 
upgrading satellite should take less time compared to before 


Additional info:

Comment 1 sandeep mutkule 2017-12-11 13:26:11 UTC
While running yum update to install/upgrade packages to 6.2 from 6.1, it creates bash scripts in /tmp/ directory like: 

/tmp/sclPliDkB 
/tmp/tmp.BGgtiKxsvJ 
/tmp/rpm-tmp.1U8aWH

these scripts contain rake tasks like: 

db:migrate
db:seed
apipie:cache:index 

For the customers having large satellite setup these tasks take long time to complete. While these tasks are running in the backend there is no user friendly message on the console, so it make user think that  " yum update" command is stuck or not responding. 

Since we are running all these above tasks while running # satellite-installer --upgrade so can we skip then while updating packages? 

If not, there should be user friendly error message on the console so admins will wait till it completes.

Comment 2 sandeep mutkule 2017-12-11 13:26:49 UTC
Attaching image from a test system for reference:

Comment 3 sandeep mutkule 2017-12-11 13:28:26 UTC
Created attachment 1366001 [details]
Image attached

Comment 6 Ashish Humbe 2017-12-13 10:07:21 UTC
Hi Brad,

Yeah, this issue is seen while 6.2 to 6.3 upgrade too. 

Here are some details from one of my recent upgrade 6.2.7 to 6.2.12 

This is the script which get created under /var/tmp/

# cat /var/tmp/rpm-tmp.dM27yT 
# We need to run the db:migrate after the install transaction
# always attempt to reencrypt after update in case new fields can be encrypted
/usr/sbin/foreman-rake db:migrate db:encrypt_all >> /var/log/foreman/db_migrate.log 2>&1 || :
/usr/sbin/foreman-rake db:seed >> /var/log/foreman/db_seed.log 2>&1 || :
/usr/sbin/foreman-rake apipie:cache:index >> /var/log/foreman/apipie_cache.log 2>&1 || :
(/sbin/service foreman status && /sbin/service foreman restart) >/dev/null 2>&1



# cat /tmp/tmp.q1hd3u46nC 
rake db:migrate db:encrypt_all

# date ; cat /tmp/tmp.1YxDGNuRI2 
Wed Dec 13 05:22:36 IST 2017
rake apipie:cache:index

# date ; cat /tmp/tmp.0WDLlgAm0y 
Wed Dec 13 05:24:14 IST 2017
rake db:migrate

# date ; cat /tmp/tmp.uugUYwHdqX  
Wed Dec 13 05:24:34 IST 2017
rake db:seed

# date ; cat /tmp/tmp.DQagGFOFML 
Wed Dec 13 05:24:58 IST 2017
rake apipie:cache:index

# date ; cat /tmp/tmp.SdPGWupI6p 
Wed Dec 13 05:26:37 IST 2017
rake db:migrate

# date ; cat /tmp/tmp.T7YSLIW9by 
Wed Dec 13 05:26:58 IST 2017
rake db:seed

# date ; cat /tmp/tmp.iNIq65b0zu 
Wed Dec 13 05:27:23 IST 2017
rake apipie:cache:index

# date ; cat /tmp/tmp.h3oe3OYPx4 
Wed Dec 13 05:29:37 IST 2017
rake db:seed

# date ; cat /tmp/tmp.LN5v3Ok0AO 
Wed Dec 13 05:30:06 IST 2017
rake ldap:refresh_usergroups

# date ; cat /tmp/tmp.tE3NdUdZ6R 
Wed Dec 13 05:30:11 IST 2017
rake apipie:cache:index

# date ; cat /tmp/tmp.WeDUj2hb4N 
Wed Dec 13 05:30:15 IST 2017
rake trends:counter

# date ; cat /tmp/tmp.tE3NdUdZ6R 
Wed Dec 13 05:30:22 IST 2017
rake apipie:cache:index

# date ; cat /tmp/tmp.UKaU3YA3Xm 
Wed Dec 13 05:32:20 IST 2017
rake db:migrate

# date ; cat /tmp/tmp.uwTR0BKM4M 
Wed Dec 13 05:32:51 IST 2017
rake db:seed

# date ; cat /tmp/tmp.OHqyik9BUh 
Wed Dec 13 05:33:11 IST 2017
rake apipie:cache

# date ; cat /tmp/tmp.OHqyik9BUh 
Wed Dec 13 05:38:19 IST 2017
rake apipie:cache

Here we can see that the yum update command was stuck for 16+ mins with no output on the console. 

I will try to get similar details for 6.2 to 6.3 upgrade and update in this bz.

Comment 7 Ivan Necas 2017-12-13 18:44:14 UTC
I'm suggesting this for foreman-maintain: the idea would be to set something (such as env variable or maybe a file present on the file-system) to skip this steps, as we will run them via the upgrade process later anyway.

This task would consist of two steps:

1. update the packaging spec to allow skipping this parts (such as here https://github.com/theforeman/foreman-packaging/blob/rpm/1.16/foreman/foreman.spec#L862), it's needed to be aware of the fact

2. teach foreman-maintain how to use the way to disable this steps, introduced in 1.

Comment 8 Ivan Necas 2017-12-13 18:59:12 UTC
Acutally, thinking more about this, while the foreman-maintain addition would be useful, as we would prevent running all the rake scripts in the rpm scripts, I think the solution might be just teaching the foreman macros how to check, if the particular rake task was already run during the transaction: this would prevent from need to be run twice.

The problem is every plugin adds the steps about db:migrate and apipie:cache into their post-trans phase. This results in running this multiple times. The solution might be:

1. in %postrans, every affected rake run would create some mark that it was run. The subsequent rake runs would notice this and would skip it's call again

2. in %pretrans, we would clear the marks from previous run to allow the posttrans to be run in case of the new update session

The mark could be achieved by a simple file, that would contain all the rake calls that were already run during the update.

Comment 9 Ivan Necas 2017-12-13 19:13:00 UTC
Created redmine issue http://projects.theforeman.org/issues/21965 from this bug

Comment 10 Bryan Kearney 2019-02-07 12:09:38 UTC
The Satellite Team is attempting to provide an accurate backlog of bugzilla requests which we feel will be resolved in the next few releases. We do not believe this bugzilla will meet that criteria, and have plans to close it out in 1 month. This is not a reflection on the validity of the request, but a reflection of the many priorities for the product. If you have any concerns about this, feel free to contact Red Hat Technical Support or your account team. If we do not hear from you, we will close this bug out. Thank you.

Comment 11 Bryan Kearney 2019-02-28 19:35:16 UTC
Thank you for your interest in Satellite 6. We have evaluated this request, and while we recognize that it is a valid request, we do not expect this to be implemented in the product in the foreseeable future. This is due to other priorities for the product, and not a reflection on the request itself. We are therefore closing this out as WONTFIX. If you have any concerns about this, please do not reopen. Instead, feel free to contact Red Hat Technical Support. Thank you.