Bug 2070993

Summary:	installer does not restart foreman.service when changing puma configuration
Product:	Red Hat Satellite	Reporter:	Brad Buckingham <bbuckingham>
Component:	Installation	Assignee:	wclark
Status:	CLOSED WONTFIX	QA Contact:	Satellite QE Team <sat-qe-bz-list>
Severity:	medium	Docs Contact:
Priority:	medium
Version:	6.10.0	CC:	ahumbe, egolov, ehelms, gtalreja, pdwyer, pmendezh, pmoravec, tasander, zhunting
Target Milestone:	Unspecified	Keywords:	Triaged
Target Release:	Unused
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	foreman-installer-2.5.2.12-1	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:	2025760	Environment:
Last Closed:	2024-04-16 13:19:08 UTC	Type:	---
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Brad Buckingham 2022-04-01 14:16:09 UTC

+++ This bug was initially created as a clone of Bug #2025760 +++

Initial Setup: puma configured and running with 4 workers and 4 workers per thread

# grep -n puma /etc/foreman-installer/scenarios.d/satellite-answers.yaml 
122:  foreman_service_puma_threads_min: 
123:  foreman_service_puma_threads_max: 4
124:  foreman_service_puma_workers: 4

# ps aux | grep -v grep | grep -e USER -e puma
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
foreman   3472  0.6  2.2 951716 462672 ?       Ssl  18:08   0:47 puma 5.3.2 (unix:///run/foreman.sock) [foreman]
foreman   4060  0.0  3.1 1141940 648704 ?      Sl   18:09   0:05 puma: cluster worker 0: 3472 [foreman]
foreman   4061  0.0  3.1 1139888 648544 ?      Sl   18:09   0:06 puma: cluster worker 1: 3472 [foreman]
foreman   4064  0.1  3.0 1119804 618152 ?      Sl   18:09   0:12 puma: cluster worker 2: 3472 [foreman]
foreman   4065  0.0  3.0 1143428 631664 ?      Sl   18:09   0:05 puma: cluster worker 3: 3472 [foreman]


Reproduction steps:

1. change puma configuration foreman_service_threads_max and/or foreman_service_puma_workers.

# satellite-installer --foreman-foreman-service-puma-workers 5 --foreman-foreman-service-puma-threads-max 5 --verbose-log-level=info

2. check # of puma worker processes running is still 4 after installer has completed:

# ps aux | grep -v grep | grep -e USER -e puma
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
foreman   8148 11.9  2.2 951776 462376 ?       Ssl  20:21   0:43 puma 5.3.2 (unix:///run/foreman.sock) [foreman]
foreman   8219  1.2  3.1 1150208 649108 ?      Sl   20:22   0:03 puma: cluster worker 0: 8148 [foreman]
foreman   8223  0.0  2.2 978532 456372 ?       Sl   20:22   0:00 puma: cluster worker 1: 8148 [foreman]
foreman   8227  0.0  2.2 978532 456260 ?       Sl   20:22   0:00 puma: cluster worker 2: 8148 [foreman]
foreman   8236  0.0  2.2 978532 456492 ?       Sl   20:22   0:00 puma: cluster worker 3: 8148 [foreman]

3. restart foreman.service: # systemctl restart foreman.service

4. Observe there are now 5 puma worker processes running:

# ps aux | grep -v grep | grep -e USER -e puma
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
foreman   8857 76.2  2.2 947880 457108 ?       Ssl  20:28   0:44 puma 5.3.2 (unix:///run/foreman.sock) [foreman]
foreman   8883  1.2  2.1 976688 450764 ?       Sl   20:28   0:00 puma: cluster worker 0: 8857 [foreman]
foreman   8886  1.1  2.1 976688 450628 ?       Sl   20:28   0:00 puma: cluster worker 1: 8857 [foreman]
foreman   8894  1.1  2.1 976688 450616 ?       Sl   20:28   0:00 puma: cluster worker 2: 8857 [foreman]
foreman   8900  1.1  2.1 976688 450956 ?       Sl   20:28   0:00 puma: cluster worker 3: 8857 [foreman]
foreman   8902  1.2  2.1 976688 450748 ?       Sl   20:28   0:00 puma: cluster worker 4: 8857 [foreman]



Further information:

You can observe that foreman.socket was restarted while foreman.service was not restarted:

2021-11-22 20:21:22 [DEBUG ] [configure] Executing: '/bin/systemctl restart -- foreman.socket'
2021-11-22 20:21:23 [INFO  ] [configure] /Service[foreman.socket]: Triggered 'refresh' from 1 event
2021-11-22 20:21:23 [DEBUG ] [configure] /Service[foreman.socket]: The container Class[Foreman::Service] will propagate my refresh event
2021-11-22 20:21:23 [DEBUG ] [configure] /Service[foreman.socket]: Evaluated in 1.36 seconds
2021-11-22 20:21:23 [DEBUG ] [configure] /Service[foreman]: Starting to evaluate the resource (1592 of 2025)
2021-11-22 20:21:23 [DEBUG ] [configure] Executing: '/bin/systemctl is-active -- foreman'
2021-11-22 20:21:23 [DEBUG ] [configure] Executing: '/bin/systemctl is-enabled -- foreman'
2021-11-22 20:21:23 [DEBUG ] [configure] Executing: '/bin/systemctl show --property=NeedDaemonReload -- foreman'
2021-11-22 20:21:23 [DEBUG ] [configure] Executing: '/bin/systemctl daemon-reload'
2021-11-22 20:21:23 [DEBUG ] [configure] Executing: '/bin/systemctl unmask -- foreman'
2021-11-22 20:21:23 [DEBUG ] [configure] Executing: '/bin/systemctl start -- foreman'
2021-11-22 20:22:08 [DEBUG ] [configure] Executing: '/bin/systemctl is-enabled -- foreman'
2021-11-22 20:22:08 [INFO  ] [configure] /Stage[main]/Foreman::Service/Service[foreman]/ensure: ensure changed 'stopped' to 'running'
2021-11-22 20:22:08 [DEBUG ] [configure] /Service[foreman]: The container Class[Foreman::Service] will propagate my refresh event
2021-11-22 20:22:08 [DEBUG ] [configure] /Service[foreman]: Unscheduling refresh on Service[foreman]
2021-11-22 20:22:08 [DEBUG ] [configure] /Service[foreman]: Evaluated in 44.66 seconds

--- Additional comment from  on 2021-11-23T02:22:08Z 

Created redmine issue https://projects.theforeman.org/issues/33973 from this bug

--- Additional comment from  on 2021-11-30T08:05:45Z 

Moving this bug to POST for triage into Satellite since the upstream issue https://projects.theforeman.org/issues/33973 has been resolved.

--- Additional comment from  on 2022-01-27T12:43:45Z 

*** Bug 2036222 has been marked as a duplicate of this bug. ***

--- Additional comment from  on 2022-02-02T14:23:35Z 

Verified on the Satellite 7.0 and snap 7 and it is working as expected     


6  satellite-installer --foreman-foreman-service-puma-workers 6 --foreman-foreman-service-puma-threads-max 6 --verbose-log-level=info
    7  systemctl status foreman.service
    8  ps aux | grep -v grep | grep -e USER -e puma
    9  systemctl restart foreman.service
   10  ps aux | grep -v grep | grep -e USER -e puma
   11  systemctl status foreman.service


● foreman.service - Foreman
   Loaded: loaded (/usr/lib/systemd/system/foreman.service; enabled; vendor preset: disabled)
  Drop-In: /etc/systemd/system/foreman.service.d
           └─installer.conf
   Active: active (running) since Wed 2022-02-02 09:20:57 EST; 39s ago
     Docs: https://theforeman.org
 Main PID: 6510 (rails)
    Tasks: 124 (limit: 126555)
   Memory: 898.7M
   CGroup: /system.slice/foreman.service
           ├─6510 puma 5.5.0 (unix:///run/foreman.sock) [foreman]
           ├─6557 puma: cluster worker 0: 6510 [foreman]
           ├─6561 puma: cluster worker 1: 6510 [foreman]
           ├─6565 puma: cluster worker 2: 6510 [foreman]
           ├─6575 puma: cluster worker 3: 6510 [foreman]
           ├─6577 puma: cluster worker 4: 6510 [foreman]
           └─6579 puma: cluster worker 5: 6510 [foreman]

Comment 3 Omkar Khatavkar 2022-04-22 09:21:31 UTC

verified with satellite 6.10 snap1  

Current Steps : 

1. [root ~]# ps aux | grep -v grep | grep -e USER -e puma
USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
foreman    65954  5.7  2.3 1077636 480384 ?      Ssl  04:59   0:40 puma 5.6.2 (unix:///run/foreman.sock) [foreman]
foreman    66099  0.5  3.2 1271588 667320 ?      Sl   04:59   0:03 puma: cluster worker 0: 65954 [foreman]
foreman    66104  0.5  3.2 1269512 668724 ?      Sl   04:59   0:03 puma: cluster worker 1: 65954 [foreman]
foreman    66110  0.7  3.2 1292200 664416 ?      Sl   04:59   0:04 puma: cluster worker 2: 65954 [foreman]

2. [root ~]# satellite-installer --foreman-foreman-service-puma-workers 8 --foreman-foreman-service-puma-threads-max 8 --verbose-log-level=info   

3. [root ~]# ps aux | grep -v grep | grep -e USER -e puma
USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
foreman    65954  5.7  2.3 1077636 480384 ?      Ssl  04:59   0:40 puma 5.6.2 (unix:///run/foreman.sock) [foreman]
foreman    66099  0.5  3.2 1271588 667320 ?      Sl   04:59   0:03 puma: cluster worker 0: 65954 [foreman]
foreman    66104  0.5  3.2 1269512 668724 ?      Sl   04:59   0:03 puma: cluster worker 1: 65954 [foreman]
foreman    66110  0.7  3.2 1292200 664416 ?      Sl   04:59   0:04 puma: cluster worker 2: 65954 [foreman]

Failing in QA

Comment 4 Evgeni Golov 2022-04-22 10:06:08 UTC

There is an attempt to explain *why* this is failing in https://bugzilla.redhat.com/show_bug.cgi?id=2025760#c6 and we should try to fix it there first, and then backport the additional fixes here.

Comment 7 Bryan Kearney 2022-04-26 16:05:29 UTC

Upstream bug assigned to egolov

Comment 8 Bryan Kearney 2022-04-26 16:05:32 UTC

Upstream bug assigned to egolov

Comment 9 Bryan Kearney 2022-04-27 20:05:18 UTC

Upstream bug assigned to wclark

Comment 11 Eric Helms 2024-04-16 13:19:08 UTC

This is a clone of a bug that was addressed in supported versions of Satellite, as 6.10 is no longer supported we are choosing not to address this issue for that release.