Note: This bug is displayed in read-only format because
the product is no longer active in Red Hat Bugzilla.
Red Hat Satellite engineering is moving the tracking of its product development work on Satellite to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "Satellite project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs will be migrated starting at the end of May. If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "Satellite project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/SAT-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Description of problem:
Pulp on Satellite 6.1.11 will start up, then its ping time as reflected by hammer ping will rapidly increase until it eventually starts producing error 500. We also verify this issue by running simple curl requests against pulp.
Version-Release number of selected component (if applicable):
Satellite 6.1.11
httpd-2.2.15-59.el6.x86_64
mod_wsgi-3.4-1.pulp.el6sat.x86_64
pulp-docker-plugins-0.2.5-1.el6_6sat.noarch
pulp-katello-0.5-1.el6_6sat.noarch
pulp-nodes-common-2.6.0.21-1.el6sat.noarch
pulp-nodes-parent-2.6.0.21-1.el6sat.noarch
pulp-puppet-plugins-2.6.0.21-1.el6sat.noarch
pulp-puppet-tools-2.6.0.21-1.el6sat.noarch
pulp-rpm-handlers-2.6.0.21-1.el6sat.noarch
pulp-rpm-plugins-2.6.0.21-1.el6sat.noarch
pulp-selinux-2.6.0.21-1.el6sat.noarch
pulp-server-2.6.0.21-1.el6sat.noarch
How reproducible:
Every time on this customer
Steps to Reproduce:
1. Start with a working Satellite system: katello-service start
2. hammer ping shows everything is working fine:
[root@tjerj348avm /]# hammer ping
[Foreman] Username:
[Foreman] Password for :
candlepin:
Status: ok
Server Response: Duration: 41ms
candlepin_auth:
Status: ok
Server Response: Duration: 42ms
pulp:
Status: ok
Server Response: Duration: 273ms
elasticsearch:
Status: ok
Server Response: Duration: 16ms
foreman_tasks:
Status: ok
Server Response: Duration: 1ms
3. Run a very simple curl request against pulp and it throws error 500.
[root@tjerj348avm /]# curl -k https://localhost/pulp/api/v2//status/
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>500 Internal Server Error</title>
</head><body>
<h1>Internal Server Error</h1>
<p>The server encountered an internal error or
misconfiguration and was unable to complete
your request.</p>
<p>Please contact the server administrator,
[no address given] and inform them of the time the error occurred,
and anything you might have done that may have
caused the error.</p>
<p>More information about this error may be available
in the server error log.</p>
</body></html>
4. Subsequent runs of hammer ping will show the 500 error.
[root@tjerj348avm /]# hammer ping
[Foreman] Username:
[Foreman] Password for :
candlepin:
Status: ok
Server Response: Duration: 83ms
candlepin_auth:
Status: ok
Server Response: Duration: 84ms
pulp:
Status: FAIL
Server Response: Message: 500 Internal Server Error
elasticsearch:
Status: ok
Server Response: Duration: 20ms
foreman_tasks:
Status: ok
Server Response: Duration: 1ms
Actual results:
Pulp will throw a 500 error within around 10-20 seconds of starting up.
If we run hammer ping repeatedly we can see pulp's ping time is consistently higher than the other services, and increasingly so.
Expected results:
Pulp should not start throwing error 500 upon requests. It should keep working and display a somewhat constant ping time.
Additional info:
Tailing the relevant logs from the moment everything works until pulp throws a 500 error shows these messages:
# tail -f /var/log/messages -f /var/log/httpd/foreman-ssl_error_ssl.log -f /var/log/mongodb/mongodb.log
==> /var/log/messages <==
Apr 27 18:48:54 tjerj348avm kernel: httpd[14343]: segfault at 10 ip 00007f3316a11889 sp 00007f33031ce180 error 4 in libpython2.6.so.1.0[7f33169b4000+15d000]
==> /var/log/httpd/foreman-ssl_error_ssl.log <==
[Thu Apr 27 18:49:07 2017] [error] [client 127.0.0.1] Premature end of script headers: webservices.wsgi
Description of problem: Pulp on Satellite 6.1.11 will start up, then its ping time as reflected by hammer ping will rapidly increase until it eventually starts producing error 500. We also verify this issue by running simple curl requests against pulp. Version-Release number of selected component (if applicable): Satellite 6.1.11 httpd-2.2.15-59.el6.x86_64 mod_wsgi-3.4-1.pulp.el6sat.x86_64 pulp-docker-plugins-0.2.5-1.el6_6sat.noarch pulp-katello-0.5-1.el6_6sat.noarch pulp-nodes-common-2.6.0.21-1.el6sat.noarch pulp-nodes-parent-2.6.0.21-1.el6sat.noarch pulp-puppet-plugins-2.6.0.21-1.el6sat.noarch pulp-puppet-tools-2.6.0.21-1.el6sat.noarch pulp-rpm-handlers-2.6.0.21-1.el6sat.noarch pulp-rpm-plugins-2.6.0.21-1.el6sat.noarch pulp-selinux-2.6.0.21-1.el6sat.noarch pulp-server-2.6.0.21-1.el6sat.noarch How reproducible: Every time on this customer Steps to Reproduce: 1. Start with a working Satellite system: katello-service start 2. hammer ping shows everything is working fine: [root@tjerj348avm /]# hammer ping [Foreman] Username: [Foreman] Password for : candlepin: Status: ok Server Response: Duration: 41ms candlepin_auth: Status: ok Server Response: Duration: 42ms pulp: Status: ok Server Response: Duration: 273ms elasticsearch: Status: ok Server Response: Duration: 16ms foreman_tasks: Status: ok Server Response: Duration: 1ms 3. Run a very simple curl request against pulp and it throws error 500. [root@tjerj348avm /]# curl -k https://localhost/pulp/api/v2//status/ <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> <html><head> <title>500 Internal Server Error</title> </head><body> <h1>Internal Server Error</h1> <p>The server encountered an internal error or misconfiguration and was unable to complete your request.</p> <p>Please contact the server administrator, [no address given] and inform them of the time the error occurred, and anything you might have done that may have caused the error.</p> <p>More information about this error may be available in the server error log.</p> </body></html> 4. Subsequent runs of hammer ping will show the 500 error. [root@tjerj348avm /]# hammer ping [Foreman] Username: [Foreman] Password for : candlepin: Status: ok Server Response: Duration: 83ms candlepin_auth: Status: ok Server Response: Duration: 84ms pulp: Status: FAIL Server Response: Message: 500 Internal Server Error elasticsearch: Status: ok Server Response: Duration: 20ms foreman_tasks: Status: ok Server Response: Duration: 1ms Actual results: Pulp will throw a 500 error within around 10-20 seconds of starting up. If we run hammer ping repeatedly we can see pulp's ping time is consistently higher than the other services, and increasingly so. Expected results: Pulp should not start throwing error 500 upon requests. It should keep working and display a somewhat constant ping time. Additional info: Tailing the relevant logs from the moment everything works until pulp throws a 500 error shows these messages: # tail -f /var/log/messages -f /var/log/httpd/foreman-ssl_error_ssl.log -f /var/log/mongodb/mongodb.log ==> /var/log/messages <== Apr 27 18:48:54 tjerj348avm kernel: httpd[14343]: segfault at 10 ip 00007f3316a11889 sp 00007f33031ce180 error 4 in libpython2.6.so.1.0[7f33169b4000+15d000] ==> /var/log/httpd/foreman-ssl_error_ssl.log <== [Thu Apr 27 18:49:07 2017] [error] [client 127.0.0.1] Premature end of script headers: webservices.wsgi