Bug 2238325
| Summary: | MaxRequestsPerChild from tuning triggers sporadic silent response for clients using HTTP/2 | ||
|---|---|---|---|
| Product: | Red Hat Satellite | Reporter: | Pavel Moravec <pmoravec> |
| Component: | Installation | Assignee: | Ewoud Kohl van Wijngaarden <ekohlvan> |
| Status: | CLOSED ERRATA | QA Contact: | Griffin Sullivan <gsulliva> |
| Severity: | high | Docs Contact: | |
| Priority: | high | ||
| Version: | 6.13.0 | CC: | ahumbe, egolov, ehelms, ekohlvan, gsulliva, momran, osousa, pcreech, pmendezh, rlavi |
| Target Milestone: | 6.15.0 | Keywords: | Triaged |
| Target Release: | Unused | ||
| Hardware: | x86_64 | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Fixed In Version: | foreman-installer-3.9.0-0 | Doc Type: | If docs needed, set a value |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2024-04-23 17:14:24 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
|
Description
Pavel Moravec
2023-09-11 13:24:27 UTC
Thanks for uncovering this. I get the impression just raising the MaxRequestsPerChild value isn't solving it. Just making it less frequent. Should we push for RHEL to include the Apache bugfix? In the mean time, we can raise the default value in our installer so it's 4000 everywhere. (In reply to Ewoud Kohl van Wijngaarden from comment #1) > Thanks for uncovering this. I get the impression just raising the > MaxRequestsPerChild value isn't solving it. Just making it less frequent. Indeed, that is my understanding as well. > Should we push for RHEL to include the Apache bugfix? In the mean time, we > can raise the default value in our installer so it's 4000 everywhere. I think pushing for RHEL fix is the right long-term way since the bug is in httpd component - let me know if I shall help with raising that BZ (i.e. preparing a standalone reproducer outside Satellite). No idea if/what some better short-to-middle term solution exists. Looking deeper at the docs we can see that for Apache there's https://httpd.apache.org/docs/2.2/mod/mpm_common.html#MaxRequestsPerChild but that's Apache 2.2, defaulting to 10000. Actually in 2.4 it was renamed to MaxConnectionsPerChild: https://httpd.apache.org/docs/2.4/mod/mpm_common.html#maxconnectionsperchild (which interprets the old MaxRequestsPerChild setting), defaulting to 0. So our default tuning doesn't limit it at all and doesn't recycle workers, but large installations do. So why do we even set this today? Back in the day with mod_wsgi and mod_passenger it was possible to have memory leaks in application code so it made sense, but now we're a pure reverse proxy with minimal code. I'd trust Apache to not leak memory and propose we drop the setting, relying on the default. Small detail: since https://github.com/puppetlabs/puppetlabs-apache/commit/cedd45b63be89ea54bd2a596e6cd3a3f60d4faf8 the parameter doesn't exist anymore and setting apache::mod::event::maxrequestsperchild in Hiera does nothing. So Foreman 3.8 includes puppetlabs-apache >= 9.0 and I hadn't noticed this before. Can you perform testing with the value set to 0 and see if it still happens? I opened https://projects.theforeman.org/issues/36784 to drop it from the newer versions (since it's pointless). If testing shows it's better to drop the tuning, we should cherry pick it further back. > Can you perform testing with the value set to 0 and see if it still happens?
I run two sets of tests:
1) MaxRequestsPerChild set to zero
2) MaxRequestsPerChild setting even removed (which should imply zero, but let double-check..)
In both cases, I run > 160k iterations / individual curl requests without an issue. So either setting prevents the bug.
Thanks for testing. The default value is 0, so both of those test cases should be the same but still good to have confirmation. The event MPM is default since Foreman 3.3 (https://projects.theforeman.org/issues/20889), so this bug affects users who chose a tuning profile on Satellite 6.12+. Moving to POST since https://github.com/theforeman/foreman-installer/commit/4462c6d4fc34cfdfe73d31c63cbf39eb979f73e6 was merged. FailedQA on 6.14 snap 19 The changes are not present in the snap. # grep maxrequestsperchild /usr/share/foreman-installer/config/foreman.hiera/tuning/sizes/*yaml /usr/share/foreman-installer/config/foreman.hiera/tuning/sizes/extra-extra-large.yaml:apache::mod::event::maxrequestsperchild: 4000 /usr/share/foreman-installer/config/foreman.hiera/tuning/sizes/extra-large.yaml:apache::mod::event::maxrequestsperchild: 4000 /usr/share/foreman-installer/config/foreman.hiera/tuning/sizes/large.yaml:apache::mod::event::maxrequestsperchild: 4000 /usr/share/foreman-installer/config/foreman.hiera/tuning/sizes/medium.yaml:apache::mod::event::maxrequestsperchild: 4000 Upstream bug assigned to ekohlvan Bulk setting Target Milestone = 6.15.0 where sat-6.15.0+ is set. Verified in 6.15.0 snap 2.1 maxrequestperchild not set in tuning profiles. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Important: Satellite 6.15.0 release), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2024:2010 The needinfo request[s] on this closed bug have been removed as they have been unresolved for 120 days |