Bug 1087844 - ISE when scheduling config channel deployment for >=400 systems in Spacewalk
Summary: ISE when scheduling config channel deployment for >=400 systems in Spacewalk
Keywords:
Status: CLOSED EOL
Alias: None
Product: Spacewalk
Classification: Community
Component: Server
Version: 2.1
Hardware: All
OS: Linux
unspecified
unspecified
Target Milestone: ---
Assignee: Michael Mráka
QA Contact: Red Hat Satellite QA List
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2014-04-15 12:40 UTC by Stephan Dühr
Modified: 2019-10-21 11:50 UTC (History)
2 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-10-21 11:50:54 UTC
Embargoed:


Attachments (Terms of Use)
Screenshot with ISE (170.82 KB, image/png)
2014-04-15 12:43 UTC, Stephan Dühr
no flags Details
Screenshot 100 Systems (187.49 KB, image/png)
2014-04-15 12:44 UTC, Stephan Dühr
no flags Details
Screenshot 200 Systems (187.04 KB, image/png)
2014-04-15 12:45 UTC, Stephan Dühr
no flags Details
Screenshot 300 Systems (188.35 KB, image/png)
2014-04-15 12:46 UTC, Stephan Dühr
no flags Details
Screenshot 400 Systems with ProxyTimeout 600 (217.98 KB, image/png)
2014-04-15 12:48 UTC, Stephan Dühr
no flags Details
pgfouine report 100 systems (247.15 KB, text/html)
2014-04-15 12:51 UTC, Stephan Dühr
no flags Details
pgfouine report 200 systems (228.19 KB, text/html)
2014-04-15 12:52 UTC, Stephan Dühr
no flags Details
pgfouine report 300 systems (208.83 KB, text/html)
2014-04-15 12:53 UTC, Stephan Dühr
no flags Details
pgfouine report 400 systems (196.20 KB, text/html)
2014-04-15 12:54 UTC, Stephan Dühr
no flags Details
pgfouine report 400 systems with ProxyTimeout 600 (199.30 KB, text/html)
2014-04-15 13:00 UTC, Stephan Dühr
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Red Hat Bugzilla 1093045 1 None None None 2021-01-20 06:05:38 UTC

Internal Links: 1093045

Description Stephan Dühr 2014-04-15 12:40:10 UTC
Description of problem:

When scheduling config file deployments for more than 400 Systems using
"Deploy all configuration files to selected subscribed systems" or
"Deploy all configuration files to all subscribed systems",
results in an ISE.

Version-Release number of selected component (if applicable):
Spacewalk 2.1 with PostgreSQL 9.2

How reproducible:

Can be reproduced by creating some hundred duplicate system 
entries in Spacewalk, it does not matter for this issue, that
the config file deployments will never be done. 

Steps to Reproduce:
3. create or use an existing config channel, and have about 10
   config files in it.
2. create 400 System Profiles in Spacewalk by running on
   a connected system:
   for i in $(seq 1 400); do rhnreg_ks --force --activationkey=<myactivationkey> ; done
   Probably a good idea to add the config channel for testing
   to the activationkey beeing used
2. in WebUI go to Configuration -> Configuration Channels,
   select the config channel for your tests, then click
   "Deploy all configuration files to all subscribed systems"
   or "Deploy all configuration files to selected subscribed systems"
   and select the systems
3. Click "Confirma & Deploy to selected Systems"

Actual results:

ISE after 210s, /var/log/httpd/ssl_error_log:
[Tue Apr 15 09:35:07 2014] [error] ajp_read_header: ajp_ilink_receive failed
[Tue Apr 15 09:35:07 2014] [error] (70007)The timeout specified has expired: proxy: read response failed from [::1]:8009 (localhost)

The 210s obviously result from
ProxyTimeout 210
in /etc/httpd/conf.d/zz-spacewalk-www.conf

Expected results:

No ISE, faster scheduling of config deployment actions

Additional info:

Increasing ProxyTimeout can resolve the ISE, however it took 373s in
my tests to complete. I'd consider increasing ProxyTimeout a workaround.
It also seems that the available resources on the Spacewalk server are
not utilized, the load stayed low while running the tests. Only one CPU
is at 100% running tomcat. Also noted that any other Browser tab, eg.
clicking on Overview or whatever, is stuck until tomcat finished adding
the config deployment actions.

I have run the tests with 100, 200, 300 and 400 Systems for
comparison, logged all SQL Queries and created pgfouine reports
that I'm going to attach.

With rising numbers of systems, the performance degrades:
# systems	ms			sys per s
100		23334			4,286
200		91130			2,195
300		200909			1,493
400		373237			1,072

Comment 1 Stephan Dühr 2014-04-15 12:43:13 UTC
Created attachment 886463 [details]
Screenshot with ISE

Comment 2 Stephan Dühr 2014-04-15 12:44:30 UTC
Created attachment 886465 [details]
Screenshot 100 Systems

Comment 3 Stephan Dühr 2014-04-15 12:45:21 UTC
Created attachment 886466 [details]
Screenshot 200 Systems

Comment 4 Stephan Dühr 2014-04-15 12:46:58 UTC
Created attachment 886468 [details]
Screenshot 300 Systems

Comment 5 Stephan Dühr 2014-04-15 12:48:54 UTC
Created attachment 886473 [details]
Screenshot 400 Systems with ProxyTimeout 600

Comment 6 Stephan Dühr 2014-04-15 12:51:42 UTC
Created attachment 886475 [details]
pgfouine report 100 systems

Comment 7 Stephan Dühr 2014-04-15 12:52:51 UTC
Created attachment 886478 [details]
pgfouine report 200 systems

Comment 8 Stephan Dühr 2014-04-15 12:53:38 UTC
Created attachment 886479 [details]
pgfouine report 300 systems

Comment 9 Stephan Dühr 2014-04-15 12:54:49 UTC
Created attachment 886480 [details]
pgfouine report 400 systems

Comment 10 Stephan Dühr 2014-04-15 13:00:25 UTC
Created attachment 886483 [details]
pgfouine report 400 systems with ProxyTimeout 600

When the ISE occurs, tomcat continues to schedule the config deployment actions, they show up under Schedule in the WebUI.

Comment 12 Michael Mráka 2019-10-21 11:50:54 UTC
Spacewalk 2.8 (and older) has already reached it's End Of Life.

Thank you for reporting this issue and we are sorry that we were not
able to fix it before end of life. If you would still like
to see this bug fixed and are able to reproduce it against current version
of Spacewalk 2.9, you are encouraged change the 'version' and re-open it.


Note You need to log in before you can comment on or make changes to this bug.