Bug 512844

Public bug summary:

Customer runs into Internal Server Error in UI and the following error in Tomcat catalina.out log when trying to add Red Hat Errata to custom channel (reproducer steps further below):
Apr 7, 2009 10:20:06 AM org.apache.jk.common.MsgAjp cpBytes
SEVERE: Buffer overflow: buffer.len=8192 pos=198 data=13580
41 42 00 02 04 01 2e 00                          | AB......
                                                 | 
                                                 | 
                                                 | 
                                                 | 
                                                 | 
                                                 | 
                                                 | 
                                                 | 
                                                 | 
                                                 | 
                                                 | 
                                                 | 
Apr 7, 2009 10:20:06 AM org.apache.jk.common.MsgAjp cpBytes
SEVERE: Overflow 
java.lang.Throwable
 at org.apache.jk.common.MsgAjp.cpBytes(MsgAjp.java:172)
 at org.apache.jk.common.MsgAjp.appendByteChunk(MsgAjp.java:146)
 at org.apache.jk.common.MsgAjp.appendBytes(MsgAjp.java:132)
 at org.apache.jk.server.JkCoyoteHandler.appendHead(JkCoyoteHandler.java:398)
 at org.apache.jk.server.JkCoyoteHandler.action(JkCoyoteHandler.java:416)
 at org.apache.coyote.Response.action(Response.java:182)
 at org.apache.coyote.Response.sendHeaders(Response.java:374)
 at org.apache.coyote.tomcat5.OutputBuffer.doFlush(OutputBuffer.java:322)
 at org.apache.coyote.tomcat5.OutputBuffer.close(OutputBuffer.java:283)
 at
org.apache.coyote.tomcat5.CoyoteResponse.finishResponse(CoyoteResponse.java:477)
 at org.apache.coyote.tomcat5.CoyoteAdapter.service(CoyoteAdapter.java:165)
 at org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:300)
 at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:374)
 at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:743)
 at
org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:675)
 at org.apache.jk.common.SocketConnection.runIt(ChannelSocket.java:866)
 at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
 at java.lang.Thread.run(Thread.java:595)
...

Steps to reproduce issue:
1. Create a new org, say, test-org;
2. Login to the new org and create a new empty custom channel, say, rhel5-i386-test1 for arch i386;
3. Go to Channel Details -> Packages -> Add for this new channel rhel5-i386-test1, and select/add all of the packages from the RHEL 5 Server i386 channel (5349 total).
4. Go to Errata -> Add -> Add Red Hat Errata for rhel5-i386-test1 channel, and select/add all of the errata from the RHEL 5 Server i386 channel (953 total). E.g., 
Package Association: Selected
Channel Version: Red Hat Enterprise Linux 5
View Associated Channels
Channel: Red Hat Enterprise Linux(v.5 for 32-bit x86)
View Associated Errata
Select All  
1 - 20 of 953 (953 selected)  
Confirm
5. Confirm.

*6. FOR CUSTOMER*: Internal Server Error and above tomcat logs (for customer, for my internal reproducer I get a different behavior from this point on, see more below). 

*6. FOR INTERNAL REPRODUCER* (with customer's db): see high java and oracle usage in top on satellite, and eventually it proceeds to the next page. "Below is a summary of the errata you have selected to clone. Contained in each CSV download is a list of errata and packages respectively that will be pushed to the channel.
Errata Summary:
Bug Fix Advisory Bug Fix Advisory:  547
Product Enhancement Advisory Product Enhancement Advisory:  136
Security Advisory Security Advisory:  270
Total Errata:  953
Download CSV
Package Summary:
i386  3210
i686  230
noarch  411
Total Packages:  3851
Download CSV"

Then I select Clone Errata, now top shows again ~99% oracle and java, before eventually the UI displaying "Connection Interrupted
The connection to the server was reset while the page was loading.
The network link was interrupted while negotiating a connection. Please
try again."

In catalina.out -

Jun 30, 2009 2:08:21 PM org.apache.catalina.startup.Catalina start
INFO: Server startup in 36860 ms
2009-06-30 14:27:52,940 [TP-Processor7] WARN 
org.apache.struts.action.RequestProcessor - Unhandled Exception thrown:
class java.lang.NumberFormatException
2009-06-30 14:27:52,977 [TP-Processor7] ERROR
com.redhat.rhn.frontend.servlets.SessionFilter - Error during transaction.
Rolling back
javax.servlet.ServletException: null
        at
org.apache.struts.action.RequestProcessor.processException(RequestProcessor.java:535)
        at
org.apache.struts.action.RequestProcessor.processActionPerform(RequestProcessor.java:433)
        at
org.apache.struts.action.RequestProcessor.process(RequestProcessor.java:236)
        at
com.redhat.rhn.frontend.struts.RhnRequestProcessor.process(RhnRequestProcessor.java:78)
        at
org.apache.struts.action.ActionServlet.process(ActionServlet.java:1196)
        at
org.apache.struts.action.ActionServlet.doGet(ActionServlet.java:414)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:743)
...
Caused by: java.lang.NumberFormatException: null
        at java.lang.Long.parseLong(Long.java:372)
        at java.lang.Long.parseLong(Long.java:461)
        at
com.redhat.rhn.frontend.action.channel.manage.AddRedHatErrataAction.execute(AddRedHatErrataAction.java:83)
        at
org.apache.struts.action.RequestProcessor.processActionPerform(RequestProcessor.java:431)
        ... 51 more

Though looks like oracle/java's still working on it -
top - 16:20:03 up  2:13,  2 users,  load average: 3.14, 3.25, 3.46
Tasks:  97 total,   4 running,  92 sleeping,   0 stopped,   1 zombie
Cpu(s): 77.7% us,  4.0% sy,  0.4% ni, 17.4% id,  0.5% wa,  0.0% hi,  0.0%
si
Mem:   2000004k total,  1736308k used,   263696k free,    29712k buffers
Swap:   557048k total,        0k used,   557048k free,  1067228k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 2339 tomcat    16   0  527m 356m  13m S 99.7 18.3  82:36.41 java
 2906 oracle    25   0  248m  86m  84m R 99.7  4.4 107:13.38 oracle

rhn-java-sat-5.1.1/code/src/com/redhat/rhn/frontend/action/channel/manage/AddRedHatErrataAction.java
...
    private static final String CID = "cid";
...
    public ActionForward execute(ActionMapping mapping,
            ActionForm formIn,
            HttpServletRequest request,
            HttpServletResponse response) {

        RequestContext requestContext = new RequestContext(request);
        User user =  requestContext.getLoggedInUser();
        Long cid = Long.parseLong(request.getParameter(CID)); <----- line
83
        Channel currentChan = ChannelFactory.lookupByIdAndUser(cid,
user);
        Channel selectedChannel = null;
...

It looks like CID "cid" in the request is null (or at least it thinks it
is)... 

Workarounds:

1. Cloning channels/errata (instead of adding) works around this problem;
2. Though I was never able to replicate the buffer overflow exception, it's possible it's related to http://www.nabble.com/is-there-a-hard-coded-size-limit-to-mod_jk-response-headers--td8350507.html
which suggested configuring the packetSize at standard coyote AJP Jk
handler on Tomcat 5.5.21 and above -
http://svn.apache.org/viewvc/tomcat/container/tc5.5.x/webapps/docs/changelog.xml?r1=483339&r2=485795&diff_format=h
 Now, on internal 5.1 satellite I see tomcat5-5.0.30-0jpp_10rh and 5.2
satellite (el5) there's tomcat5-5.5.23-0jpp.7.el5_2.1 so looks like this
is a configuration we can try out on satellite 5.2 or 5.3 beta as a
possible tuning/workaround conf change.

For Engineering:

More details and reproducer info included in internal updates, if you need anything else, let me know. Thanks.

So flipping this to wont_fix as per discussion.  If there is any issue with this, please re-open.

Thanks

Reopening, more than one customer hitting it, so going to investigate further.

so this shouldn't be reproducible on 5.3, but an inefficient query will make this difficult to try to reproduce using steps in comment #1.  

This was fixed in spacewalk master in:
914db78b3a4f2a805e16914f62660bb7438ec11b

So this bug is actually present in 5.3.0 GA (and thus is not caused by this fix).  There are two ways to get past this bug:

a) use a cloned channel (not a plain custom channel)
b) Click the "View Associated Errata" button before selecting the errata and hitting confirm

This is fixed in spacewalk.  


Not sure if we want to fix this for this errata.  I'll let cliff chime in...

Jiri, This is quite odd.  It looked like perl-DateTime was not installed and that's what was causing the issue (I think).

I installed that package on that box and the perl pages seem to be working.

-Justin

rhel4 platform also ok, no error during cloning errata

An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHBA-2010-0021.html