| Summary: | csprocessor push fails with ClientResponseFailure | ||
|---|---|---|---|
| Product: | [Community] PressGang CCMS | Reporter: | Stephen Gordon <sgordon> |
| Component: | CSProcessor | Assignee: | pressgang-ccms-dev |
| Status: | CLOSED CURRENTRELEASE | QA Contact: | |
| Severity: | high | Docs Contact: | |
| Priority: | high | ||
| Version: | 1.1 | CC: | lnewson, mcaspers |
| Target Milestone: | --- | ||
| Target Release: | --- | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | Bug Fix | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2014-02-23 23:40:25 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Bug Depends On: | |||
| Bug Blocks: | 1013825 | ||
|
Comment 2
Matthew Casperson
2013-10-01 21:16:08 UTC
Rebooted server and tested an edit to a large spec. Is working ok now. Specs are taking a while to process through: steveg> Saving to the server... 8<steveg> Waiting for a response from the server... 8<steveg> Waiting for a response from the server... 8<steveg> Waiting for a response from the server... 8<steveg> Waiting for a response from the server... 8<steveg> Waiting for a response from the server... 8<steveg> Waiting for a response from the server... 8<steveg> Waiting for a response from the server... 8<steveg> Waiting for a response from the server... 8<steveg> Waiting for a response from the server... 8<steveg> Waiting for a response from the server... 8<steveg> Waiting for a response from the server... 8<steveg> ^CWaiting for a response from the server... 8<steveg> WARN: Line 0: The Content Specification has revision topics which can't have tags added, so the tags will be ignored for those topics. 8<steveg> -> [RHOS 4.0, Writer = Unknown] 8<steveg> INFO: The Content Specification is valid. 8<steveg> INFO: The Content Specification saved successfully. 8<steveg> Content Specification ID: 15807 8<steveg> Revision: 542381 From the looks of the log, it took over 5 minutes to process the spec. This is possibly due to other background processes running at the same time. This caused a transaction timeout, which then prevented any changes from being saved. I have increased the timeout and added a connection validation checker to the server config. There is no reason why that spec should take that long to process as far as I can see (it only has about 400 nodes), so there is likely something wrong in the processing. Also updating just those values will make no difference to the reason a 502 error is being returned, as that is caused by the HTTP timeout which is currently set at 3 mins (note the 18 waiting messages in the first comment). In saying that though it will help to mitigate the Transaction reaper activating. This may have been related to an issue that came up this morning with JavaMelody. The full explaination is documented at http://stackoverflow.com/questions/19147762/javamelody-crashing-the-server-with-thousands-of-rrd-files/19147763#19147763, but the short story is that JavaMelody needs some special configuration when used to monitor a REST API. We have disabled JavaMelody for now on the production server, with an eye to aggregating the files (currently in testing on the dev server). We'll see if this has any impact on the performance issues and timeouts. Added the settings
<context-param>
<param-name>javamelody.http-transform-pattern</param-name>
<param-value>(?<=/rest/).*</param-value>
</context-param>
<context-param>
<param-name>javamelody.sql-transform-pattern</param-name>
<param-value>.*</param-value>
</context-param>
This results in 32 files being created in /tmp/javamelody after a couple of topic and spec edits.
Prior to that, just booting the REST server resulted in over 100 files being created, and that number quickly grew to over 1000.
I'll do some more testing on the dev server to make sure these settings don't have any other side effects, but it looks like they will fix the problem.
Ok, we're still having problems with JavaMelody disabled. So this looks like an issue with open file limits, like in https://community.jboss.org/thread/213611?start=0&tstart=0. The open file limit has been increased to 4096. The cause of exceeding the file limit was the way the REST API version was being retrieved. The version was being retrieved from a value set in mainfest.mf at compile time and then added as a HTTP Header to all responses during runtime. The problem here is that it was doing this for every request and due to a bug where the InputStream that opened the manifest.mf was not being closed, a lot of file handles were being left open until the Garbage collector ran. As such I've done two things to fix that issue. The first is to ensure that the InputStream is closed and the second is to cache the version so that it doesn't have to keep being looked up all the time. Hours apply to 2013-10-01 Marking as closed as this should have been fixed some time ago. If the issue arises again please reopen this bug. |