2070993 – installer does not restart foreman.service when changing puma configuration

Red Hat Satellite engineering is moving the tracking of its product development work on Satellite to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "Satellite project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs will be migrated starting at the end of May. If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "Satellite project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/SAT-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 2070993 - installer does not restart foreman.service when changing puma configuration

Summary: installer does not restart foreman.service when changing puma configuration

Keywords:
Status:	CLOSED WONTFIX
Alias:	None
Product:	Red Hat Satellite
Classification:	Red Hat
Component:	Installation
Sub Component:
Version:	6.10.0
Hardware:	Unspecified
OS:	Unspecified
Priority:	medium
Severity:	medium
Target Milestone:	Unspecified
Assignee:	wclark
QA Contact:	Satellite QE Team
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2022-04-01 14:16 UTC by Brad Buckingham
Modified:	2024-04-16 13:20 UTC (History)
CC List:	9 users (show)
Fixed In Version:	foreman-installer-2.5.2.12-1
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:	2025760
Environment:
Last Closed:	2024-04-16 13:19:08 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)

Links
System	ID	Priority	Status	Summary	Last Updated
Foreman Issue Tracker	33973	Normal	Closed	installer does not restart foreman.service when changing puma configuration	2022-04-01 14:16:21 UTC
Foreman Issue Tracker	34824	Normal	Closed	Installer does not restart foreman.service when changing Puma configuration	2022-06-02 16:00:44 UTC
Red Hat Issue Tracker	SAT-24593	None	None	None	2024-04-16 13:20:30 UTC

Description Brad Buckingham 2022-04-01 14:16:09 UTC

+++ This bug was initially created as a clone of Bug #2025760 +++

Initial Setup: puma configured and running with 4 workers and 4 workers per thread

# grep -n puma /etc/foreman-installer/scenarios.d/satellite-answers.yaml 
122:  foreman_service_puma_threads_min: 
123:  foreman_service_puma_threads_max: 4
124:  foreman_service_puma_workers: 4

# ps aux | grep -v grep | grep -e USER -e puma
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
foreman   3472  0.6  2.2 951716 462672 ?       Ssl  18:08   0:47 puma 5.3.2 (unix:///run/foreman.sock) [foreman]
foreman   4060  0.0  3.1 1141940 648704 ?      Sl   18:09   0:05 puma: cluster worker 0: 3472 [foreman]
foreman   4061  0.0  3.1 1139888 648544 ?      Sl   18:09   0:06 puma: cluster worker 1: 3472 [foreman]
foreman   4064  0.1  3.0 1119804 618152 ?      Sl   18:09   0:12 puma: cluster worker 2: 3472 [foreman]
foreman   4065  0.0  3.0 1143428 631664 ?      Sl   18:09   0:05 puma: cluster worker 3: 3472 [foreman]


Reproduction steps:

1. change puma configuration foreman_service_threads_max and/or foreman_service_puma_workers.

# satellite-installer --foreman-foreman-service-puma-workers 5 --foreman-foreman-service-puma-threads-max 5 --verbose-log-level=info

2. check # of puma worker processes running is still 4 after installer has completed:

# ps aux | grep -v grep | grep -e USER -e puma
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
foreman   8148 11.9  2.2 951776 462376 ?       Ssl  20:21   0:43 puma 5.3.2 (unix:///run/foreman.sock) [foreman]
foreman   8219  1.2  3.1 1150208 649108 ?      Sl   20:22   0:03 puma: cluster worker 0: 8148 [foreman]
foreman   8223  0.0  2.2 978532 456372 ?       Sl   20:22   0:00 puma: cluster worker 1: 8148 [foreman]
foreman   8227  0.0  2.2 978532 456260 ?       Sl   20:22   0:00 puma: cluster worker 2: 8148 [foreman]
foreman   8236  0.0  2.2 978532 456492 ?       Sl   20:22   0:00 puma: cluster worker 3: 8148 [foreman]

3. restart foreman.service: # systemctl restart foreman.service

4. Observe there are now 5 puma worker processes running:

# ps aux | grep -v grep | grep -e USER -e puma
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
foreman   8857 76.2  2.2 947880 457108 ?       Ssl  20:28   0:44 puma 5.3.2 (unix:///run/foreman.sock) [foreman]
foreman   8883  1.2  2.1 976688 450764 ?       Sl   20:28   0:00 puma: cluster worker 0: 8857 [foreman]
foreman   8886  1.1  2.1 976688 450628 ?       Sl   20:28   0:00 puma: cluster worker 1: 8857 [foreman]
foreman   8894  1.1  2.1 976688 450616 ?       Sl   20:28   0:00 puma: cluster worker 2: 8857 [foreman]
foreman   8900  1.1  2.1 976688 450956 ?       Sl   20:28   0:00 puma: cluster worker 3: 8857 [foreman]
foreman   8902  1.2  2.1 976688 450748 ?       Sl   20:28   0:00 puma: cluster worker 4: 8857 [foreman]



Further information:

You can observe that foreman.socket was restarted while foreman.service was not restarted:

2021-11-22 20:21:22 [DEBUG ] [configure] Executing: '/bin/systemctl restart -- foreman.socket'
2021-11-22 20:21:23 [INFO  ] [configure] /Service[foreman.socket]: Triggered 'refresh' from 1 event
2021-11-22 20:21:23 [DEBUG ] [configure] /Service[foreman.socket]: The container Class[Foreman::Service] will propagate my refresh event
2021-11-22 20:21:23 [DEBUG ] [configure] /Service[foreman.socket]: Evaluated in 1.36 seconds
2021-11-22 20:21:23 [DEBUG ] [configure] /Service[foreman]: Starting to evaluate the resource (1592 of 2025)
2021-11-22 20:21:23 [DEBUG ] [configure] Executing: '/bin/systemctl is-active -- foreman'
2021-11-22 20:21:23 [DEBUG ] [configure] Executing: '/bin/systemctl is-enabled -- foreman'
2021-11-22 20:21:23 [DEBUG ] [configure] Executing: '/bin/systemctl show --property=NeedDaemonReload -- foreman'
2021-11-22 20:21:23 [DEBUG ] [configure] Executing: '/bin/systemctl daemon-reload'
2021-11-22 20:21:23 [DEBUG ] [configure] Executing: '/bin/systemctl unmask -- foreman'
2021-11-22 20:21:23 [DEBUG ] [configure] Executing: '/bin/systemctl start -- foreman'
2021-11-22 20:22:08 [DEBUG ] [configure] Executing: '/bin/systemctl is-enabled -- foreman'
2021-11-22 20:22:08 [INFO  ] [configure] /Stage[main]/Foreman::Service/Service[foreman]/ensure: ensure changed 'stopped' to 'running'
2021-11-22 20:22:08 [DEBUG ] [configure] /Service[foreman]: The container Class[Foreman::Service] will propagate my refresh event
2021-11-22 20:22:08 [DEBUG ] [configure] /Service[foreman]: Unscheduling refresh on Service[foreman]
2021-11-22 20:22:08 [DEBUG ] [configure] /Service[foreman]: Evaluated in 44.66 seconds

--- Additional comment from  on 2021-11-23T02:22:08Z 

Created redmine issue https://projects.theforeman.org/issues/33973 from this bug

--- Additional comment from  on 2021-11-30T08:05:45Z 

Moving this bug to POST for triage into Satellite since the upstream issue https://projects.theforeman.org/issues/33973 has been resolved.

--- Additional comment from  on 2022-01-27T12:43:45Z 

*** Bug 2036222 has been marked as a duplicate of this bug. ***

--- Additional comment from  on 2022-02-02T14:23:35Z 

Verified on the Satellite 7.0 and snap 7 and it is working as expected     


6  satellite-installer --foreman-foreman-service-puma-workers 6 --foreman-foreman-service-puma-threads-max 6 --verbose-log-level=info
    7  systemctl status foreman.service
    8  ps aux | grep -v grep | grep -e USER -e puma
    9  systemctl restart foreman.service
   10  ps aux | grep -v grep | grep -e USER -e puma
   11  systemctl status foreman.service


● foreman.service - Foreman
   Loaded: loaded (/usr/lib/systemd/system/foreman.service; enabled; vendor preset: disabled)
  Drop-In: /etc/systemd/system/foreman.service.d
           └─installer.conf
   Active: active (running) since Wed 2022-02-02 09:20:57 EST; 39s ago
     Docs: https://theforeman.org
 Main PID: 6510 (rails)
    Tasks: 124 (limit: 126555)
   Memory: 898.7M
   CGroup: /system.slice/foreman.service
           ├─6510 puma 5.5.0 (unix:///run/foreman.sock) [foreman]
           ├─6557 puma: cluster worker 0: 6510 [foreman]
           ├─6561 puma: cluster worker 1: 6510 [foreman]
           ├─6565 puma: cluster worker 2: 6510 [foreman]
           ├─6575 puma: cluster worker 3: 6510 [foreman]
           ├─6577 puma: cluster worker 4: 6510 [foreman]
           └─6579 puma: cluster worker 5: 6510 [foreman]

Comment 3 Omkar Khatavkar 2022-04-22 09:21:31 UTC

verified with satellite 6.10 snap1  

Current Steps : 

1. [root ~]# ps aux | grep -v grep | grep -e USER -e puma
USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
foreman    65954  5.7  2.3 1077636 480384 ?      Ssl  04:59   0:40 puma 5.6.2 (unix:///run/foreman.sock) [foreman]
foreman    66099  0.5  3.2 1271588 667320 ?      Sl   04:59   0:03 puma: cluster worker 0: 65954 [foreman]
foreman    66104  0.5  3.2 1269512 668724 ?      Sl   04:59   0:03 puma: cluster worker 1: 65954 [foreman]
foreman    66110  0.7  3.2 1292200 664416 ?      Sl   04:59   0:04 puma: cluster worker 2: 65954 [foreman]

2. [root ~]# satellite-installer --foreman-foreman-service-puma-workers 8 --foreman-foreman-service-puma-threads-max 8 --verbose-log-level=info   

3. [root ~]# ps aux | grep -v grep | grep -e USER -e puma
USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
foreman    65954  5.7  2.3 1077636 480384 ?      Ssl  04:59   0:40 puma 5.6.2 (unix:///run/foreman.sock) [foreman]
foreman    66099  0.5  3.2 1271588 667320 ?      Sl   04:59   0:03 puma: cluster worker 0: 65954 [foreman]
foreman    66104  0.5  3.2 1269512 668724 ?      Sl   04:59   0:03 puma: cluster worker 1: 65954 [foreman]
foreman    66110  0.7  3.2 1292200 664416 ?      Sl   04:59   0:04 puma: cluster worker 2: 65954 [foreman]

Failing in QA

Comment 4 Evgeni Golov 2022-04-22 10:06:08 UTC

There is an attempt to explain *why* this is failing in https://bugzilla.redhat.com/show_bug.cgi?id=2025760#c6 and we should try to fix it there first, and then backport the additional fixes here.

Comment 7 Bryan Kearney 2022-04-26 16:05:29 UTC

Upstream bug assigned to egolov

Comment 8 Bryan Kearney 2022-04-26 16:05:32 UTC

Upstream bug assigned to egolov

Comment 9 Bryan Kearney 2022-04-27 20:05:18 UTC

Upstream bug assigned to wclark

Comment 11 Eric Helms 2024-04-16 13:19:08 UTC

This is a clone of a bug that was addressed in supported versions of Satellite, as 6.10 is no longer supported we are choosing not to address this issue for that release.

Note You need to log in before you can comment on or make changes to this bug.