Red Hat Satellite engineering is moving the tracking of its product development work on Satellite to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "Satellite project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs will be migrated starting at the end of May. If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "Satellite project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/SAT-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1953416 - Foreman proxy request timed out when batch triggering remote tasks take more than 1 minutes to trigger.
Summary: Foreman proxy request timed out when batch triggering remote tasks take more...
Keywords:
Status: CLOSED WORKSFORME
Alias: None
Product: Red Hat Satellite
Classification: Red Hat
Component: Foreman Proxy
Version: 6.8.0
Hardware: Unspecified
OS: Unspecified
unspecified
medium
Target Milestone: Unspecified
Assignee: Lukas Zapletal
QA Contact: Satellite QE Team
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-04-26 04:21 UTC by Hao Chang Yu
Modified: 2023-06-14 12:03 UTC (History)
2 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2023-06-14 12:03:11 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Foreman Issue Tracker 32418 0 Normal Ready For Testing Introduce foreman request timeout 2021-04-26 06:13:25 UTC

Description Hao Chang Yu 2021-04-26 04:21:32 UTC
Description of problem:
The connection between Foreman Proxy and the Smart_proxy_dynflow_core is timed out(60 seconds) even we increased the "proxy_request_timeout" setting from the Satellite. It is because the setting is for the connection between Foreman and the Foreman Proxy. The connection between Foreman Proxy and Smart_proxy_dynflow_core is still 60 seconds (Net::HTTP default). 


Steps to Reproduce:
1. Satellite Web UI Administer -> Settings -> General tab -> Capsule request timeout -> Set to 600
2. Ensure foreman tasks batch triggering is On.
3. To simulate a task taking long time to trigger. Hack the following code to sleep for 300 seconds.

In "/opt/theforeman/tfm/root/usr/share/gems/gems/smart_proxy_dynflow_core-0.2.6/lib/smart_proxy_dynflow_core/api.rb"
    post "/tasks/launch/?" do
      sleep 300    <============== Add this line
      params = MultiJson.load(request.body.read)
      launcher = launcher_class(params).new(world, callback_host(params, request), params.fetch('options', {}))
      launcher.launch!(params['input'])
      launcher.results.to_json
    end

4. Restart foreman-proxy

systemctl restart foreman-proxy

5. Trigger any REX job against multiple hosts.

Actual results:
Parent task gets 500 Internal Server Error and the hosts' job hang forever.

Expected results:
No timed out occur. Add a setting to adjust the foreman-proxy timeout as needed.


Additional info:

# Dynflow parent Task
Error:
RestClient::InternalServerError
500 Internal Server Error
---
- "/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.2/lib/restclient/abstract_response.rb:223:in
  `exception_with_response'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.2/lib/restclient/abstract_response.rb:103:in
  `return!'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.2/lib/restclient/request.rb:809:in
  `process_result'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.2/lib/restclient/request.rb:725:in
  `block in transmit'"
- "/opt/rh/rh-ruby25/root/usr/share/ruby/net/http.rb:910:in `start'"


# proxy.log
2021-04-26T13:47:14 d487df31 [I] Started POST /dynflow/tasks/launch 
2021-04-26T13:47:14 d487df31 [D] verifying remote client hao-satellite66.usersys.redhat.com (based on SSL_CLIENT_CERT) against trusted_hosts ["hao-satellite66.usersys.redhat.com"]
2021-04-26T13:47:14 d487df31 [D] Proxy request from hao-satellite66.usersys.redhat.com:9090/dynflow/tasks/launch to https://hao-satellite66.usersys.redhat.com:8008/tasks/launch
2021-04-26T13:48:14 d487df31 [W] Error processing request 'd487df31-a370-49eb-9971-d96ad46c4235: <Net::ReadTimeout>: Net::ReadTimeout
/opt/rh/rh-ruby25/root/usr/share/ruby/net/protocol.rb:181:in `rbuf_fill'
/opt/rh/rh-ruby25/root/usr/share/ruby/net/protocol.rb:157:in `readuntil'
/opt/rh/rh-ruby25/root/usr/share/ruby/net/protocol.rb:167:in `readline'
/opt/rh/rh-ruby25/root/usr/share/ruby/net/http/response.rb:40:in `read_status_line'
/opt/rh/rh-ruby25/root/usr/share/ruby/net/http/response.rb:29:in `read_new'
/opt/rh/rh-ruby25/root/usr/share/ruby/net/http.rb:1497:in `block in transport_request'
/opt/rh/rh-ruby25/root/usr/share/ruby/net/http.rb:1494:in `catch'
/opt/rh/rh-ruby25/root/usr/share/ruby/net/http.rb:1494:in `transport_request'
/opt/rh/rh-ruby25/root/usr/share/ruby/net/http.rb:1467:in `request'
/opt/rh/rh-ruby25/root/usr/share/ruby/net/http.rb:1460:in `block in request'
/opt/rh/rh-ruby25/root/usr/share/ruby/net/http.rb:910:in `start'
/opt/rh/rh-ruby25/root/usr/share/ruby/net/http.rb:1458:in `request'
/usr/share/foreman-proxy/lib/proxy/request.rb:48:in `send_request'
...

# Smart proxy dynflow core log
[26/Apr/2021:13:47:14 AEST] "POST /tasks/launch? HTTP/1.1" 200 0  <==== return 0 body length

Comment 1 Lukas Zapletal 2021-04-26 06:13:28 UTC
The setting only controls Foreman -> Foreman proxy communication and not the other direction.

Created a patch: https://projects.theforeman.org/issues/32418

Comment 2 Adam Ruzicka 2023-06-14 12:03:11 UTC
While the 60 second timeout is still there, newer satellites contain a couple of fixes (removal of smart proxy dynflow core, async batch triggering) that should prevent us from hitting the timeout. I'll go ahead and close this as the fixes mentioned previously should be good enough. If you feel the fixes are not sufficient and still manage to hit the timeouts, feel free to reopen and we can revisit this.


Note You need to log in before you can comment on or make changes to this bug.