Bug 1907801

This is going to vary system to system. If you can point to a particular service taking longer than it ought to we can investigate this. Else wise I would ask that this be closed not a bug.

I have seen this problem on all the 6.9 setups, dynflow-sidekiq@orchestrator , foreman, puppetserver, dynflow-sidekiq@worker, dynflow-sidekiq@worker-hosts-queue services start takes a bit longer.

On the upgraded setup,  satellite service restart took 12 minutes, I think some, this is something unexpected.

# date; foreman-maintain service restart; date
Fri Dec 18 05:04:50 EST 2020
Running Restart Services
================================================================================
Check if command is run as root user:                                 [OK]
--------------------------------------------------------------------------------
Restart applicable services: 

Stopping the following service(s):
rh-mongodb34-mongod, postgresql, qdrouterd, qpidd, rh-redis5-redis, squid, pulp_celerybeat, pulp_resource_manager, pulp_streamer, pulp_workers, smart_proxy_dynflow_core, tomcat, dynflow-sidekiq@orchestrator, elasticsearch, foreman, httpd, puppetserver, dynflow-sidekiq@worker, dynflow-sidekiq@worker-hosts-queue, foreman-proxy
\ stopping foreman                                                              
Warning: Stopping foreman.service, but it can still be activated by:
  foreman.socket
- All services stopped                                                          

Starting the following service(s):
rh-mongodb34-mongod, postgresql, qdrouterd, qpidd, rh-redis5-redis, squid, pulp_celerybeat, pulp_resource_manager, pulp_streamer, pulp_workers, smart_proxy_dynflow_core, tomcat, dynflow-sidekiq@orchestrator, elasticsearch, foreman, httpd, puppetserver, dynflow-sidekiq@worker, dynflow-sidekiq@worker-hosts-queue, foreman-proxy
/ starting dynflow-sidekiq@orchestrator                                         
\ starting dynflow-sidekiq@orchestrator                                         
/ starting dynflow-sidekiq@orchestrator                                         
| starting dynflow-sidekiq@orchestrator                                         
\ All services started                                                [OK]      
--------------------------------------------------------------------------------

Fri Dec 18 05:16:00 EST 2020

#

After some digging, it was found that on slower machines any service or action that loads the underlying Rails application stack can take a long time (~2 minutes) on the reproducer machine. This can be mitigated through:

 * changing foreman-maintain service actions to happen in parallel given systemd can handle all this
 * investigating speed ups to the Rails loading process

Therefore, this BZ has been modified to focus on the foreman-maintain improvements. An additional bug has been created as a clone (https://bugzilla.redhat.com/show_bug.cgi?id=1917883) to handle future investigations into loading the stack quicker.

Created redmine issue https://projects.theforeman.org/issues/31680 from this bug

Hello Devendra,

I have raised PR[1] and I can see 1 minute faster restart. Can you give it a try?
[1] https://github.com/theforeman/foreman_maintain/pull/446

Thank You,
Amit Upadhye.

Adding Need Info on Devendra.

(In reply to Amit Upadhye from comment #9)
> Hello Devendra,
> 
> I have raised PR[1] and I can see 1 minute faster restart. Can you give it a
> try?
> [1] https://github.com/theforeman/foreman_maintain/pull/446
> 
> Thank You,
> Amit Upadhye.

Hi Amit, 

I have verified your PR on 6.9 Snap 12 and observed the great performance improvement. In 6.9 Snap 9, the service restart took ~ 12 minutes on the upgrade box, and now it takes 5 minutes.

# date; foreman-maintain service restart;date
Thu Feb 11 23:36:16 EST 2021
Running Restart Services
================================================================================
Check if command is run as root user:                                 [OK]
--------------------------------------------------------------------------------
Restart applicable services: 

Stopping the following service(s):
rh-mongodb34-mongod, postgresql, qdrouterd, qpidd, rh-redis5-redis, squid, pulp_celerybeat, pulp_resource_manager, pulp_streamer, pulp_workers, smart_proxy_dynflow_core, tomcat, dynflow-sidekiq@orchestrator, elasticsearch, foreman, httpd, puppetserver, dynflow-sidekiq@worker, dynflow-sidekiq@worker-hosts-queue, foreman-proxy
\ All services stopped                                                          

Starting the following service(s):
rh-mongodb34-mongod, postgresql, qdrouterd, qpidd, rh-redis5-redis, squid, pulp_celerybeat, pulp_resource_manager, pulp_streamer, pulp_workers, smart_proxy_dynflow_core, tomcat, dynflow-sidekiq@orchestrator, elasticsearch, foreman, httpd, puppetserver, dynflow-sidekiq@worker, dynflow-sidekiq@worker-hosts-queue, foreman-proxy
- All services started                                                [OK]      
--------------------------------------------------------------------------------

Thu Feb 11 23:41:45 EST 2021
# 

I have tested this PR on the newly installed satellite too and saw the performance improvement also there. In 6.9 Snap9 the service restart took 4 to 5 minutes but now it takes 2 to 3 minutes.


# date; foreman-maintain service restart;date
Fri Feb 12 00:21:08 EST 2021
Running Restart Services
================================================================================
Check if command is run as root user:                                 [OK]
--------------------------------------------------------------------------------
Restart applicable services: 

Stopping the following service(s):
rh-mongodb34-mongod, postgresql, qdrouterd, qpidd, rh-redis5-redis, squid, pulp_celerybeat, pulp_resource_manager, pulp_streamer, pulp_workers, smart_proxy_dynflow_core, tomcat, dynflow-sidekiq@orchestrator, foreman, httpd, puppetserver, dynflow-sidekiq@worker, dynflow-sidekiq@worker-hosts-queue, foreman-proxy
| All services stopped                                                          

Starting the following service(s):
rh-mongodb34-mongod, postgresql, qdrouterd, qpidd, rh-redis5-redis, squid, pulp_celerybeat, pulp_resource_manager, pulp_streamer, pulp_workers, smart_proxy_dynflow_core, tomcat, dynflow-sidekiq@orchestrator, foreman, httpd, puppetserver, dynflow-sidekiq@worker, dynflow-sidekiq@worker-hosts-queue, foreman-proxy
\ All services started                                                [OK]      
--------------------------------------------------------------------------------

Fri Feb 12 00:23:45 EST 2021

Moving this bug to POST for triage into Satellite since the upstream issue https://projects.theforeman.org/issues/31680 has been resolved.

This breaks restarting of pulpcore-worker@* services.  This service needs --all passed to it for both start and stop and it doesn't seem like the change does that?

Amit, this particular code:

https://github.com/theforeman/foreman_maintain/blob/master/lib/foreman_maintain/utils/service/systemd.rb#L23 

is what handled this along side:

https://github.com/theforeman/foreman_maintain/blob/master/definitions/features/pulpcore.rb#L16


Maybe we could look for that all option and start those separately ?

Moving it to assigned state to fix concern mentioned in comment 15

Amit, looks like https://github.com/theforeman/foreman_maintain/pull/458/files got merged, should this be in POST?

Verified.

Tested on: 
Satellite 6.9.0 Snap 19.1
Version: rubygem-foreman_maintain-0.7.8-1.el7sat.noarch

Steps:
# time foreman-maintain service restart

Observations:
- On the newly installed 6.9.0 Satellite/Capsule, it takes 2 to 3 minutes for service restart.
- On upgrade setup of 6.9.0 Satellite/Capsule, it takes 4 to 5 minutes for service restart. 
- On 6.9.0 Satellite with pulpcore services, all pulpcore-worker@* services are being restarted and it takes 2 to 3 minutes for service restart.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Satellite 6.9 Satellite Maintenance Release), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2021:1312