Escalated to Bugzilla from IssueTracker
Public bug summary: Customer runs into Internal Server Error in UI and the following error in Tomcat catalina.out log when trying to add Red Hat Errata to custom channel (reproducer steps further below): Apr 7, 2009 10:20:06 AM org.apache.jk.common.MsgAjp cpBytes SEVERE: Buffer overflow: buffer.len=8192 pos=198 data=13580 41 42 00 02 04 01 2e 00 | AB...... | | | | | | | | | | | | Apr 7, 2009 10:20:06 AM org.apache.jk.common.MsgAjp cpBytes SEVERE: Overflow java.lang.Throwable at org.apache.jk.common.MsgAjp.cpBytes(MsgAjp.java:172) at org.apache.jk.common.MsgAjp.appendByteChunk(MsgAjp.java:146) at org.apache.jk.common.MsgAjp.appendBytes(MsgAjp.java:132) at org.apache.jk.server.JkCoyoteHandler.appendHead(JkCoyoteHandler.java:398) at org.apache.jk.server.JkCoyoteHandler.action(JkCoyoteHandler.java:416) at org.apache.coyote.Response.action(Response.java:182) at org.apache.coyote.Response.sendHeaders(Response.java:374) at org.apache.coyote.tomcat5.OutputBuffer.doFlush(OutputBuffer.java:322) at org.apache.coyote.tomcat5.OutputBuffer.close(OutputBuffer.java:283) at org.apache.coyote.tomcat5.CoyoteResponse.finishResponse(CoyoteResponse.java:477) at org.apache.coyote.tomcat5.CoyoteAdapter.service(CoyoteAdapter.java:165) at org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:300) at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:374) at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:743) at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:675) at org.apache.jk.common.SocketConnection.runIt(ChannelSocket.java:866) at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684) at java.lang.Thread.run(Thread.java:595) ... Steps to reproduce issue: 1. Create a new org, say, test-org; 2. Login to the new org and create a new empty custom channel, say, rhel5-i386-test1 for arch i386; 3. Go to Channel Details -> Packages -> Add for this new channel rhel5-i386-test1, and select/add all of the packages from the RHEL 5 Server i386 channel (5349 total). 4. Go to Errata -> Add -> Add Red Hat Errata for rhel5-i386-test1 channel, and select/add all of the errata from the RHEL 5 Server i386 channel (953 total). E.g., Package Association: Selected Channel Version: Red Hat Enterprise Linux 5 View Associated Channels Channel: Red Hat Enterprise Linux(v.5 for 32-bit x86) View Associated Errata Select All 1 - 20 of 953 (953 selected) Confirm 5. Confirm. *6. FOR CUSTOMER*: Internal Server Error and above tomcat logs (for customer, for my internal reproducer I get a different behavior from this point on, see more below). *6. FOR INTERNAL REPRODUCER* (with customer's db): see high java and oracle usage in top on satellite, and eventually it proceeds to the next page. "Below is a summary of the errata you have selected to clone. Contained in each CSV download is a list of errata and packages respectively that will be pushed to the channel. Errata Summary: Bug Fix Advisory Bug Fix Advisory: 547 Product Enhancement Advisory Product Enhancement Advisory: 136 Security Advisory Security Advisory: 270 Total Errata: 953 Download CSV Package Summary: i386 3210 i686 230 noarch 411 Total Packages: 3851 Download CSV" Then I select Clone Errata, now top shows again ~99% oracle and java, before eventually the UI displaying "Connection Interrupted The connection to the server was reset while the page was loading. The network link was interrupted while negotiating a connection. Please try again." In catalina.out - Jun 30, 2009 2:08:21 PM org.apache.catalina.startup.Catalina start INFO: Server startup in 36860 ms 2009-06-30 14:27:52,940 [TP-Processor7] WARN org.apache.struts.action.RequestProcessor - Unhandled Exception thrown: class java.lang.NumberFormatException 2009-06-30 14:27:52,977 [TP-Processor7] ERROR com.redhat.rhn.frontend.servlets.SessionFilter - Error during transaction. Rolling back javax.servlet.ServletException: null at org.apache.struts.action.RequestProcessor.processException(RequestProcessor.java:535) at org.apache.struts.action.RequestProcessor.processActionPerform(RequestProcessor.java:433) at org.apache.struts.action.RequestProcessor.process(RequestProcessor.java:236) at com.redhat.rhn.frontend.struts.RhnRequestProcessor.process(RhnRequestProcessor.java:78) at org.apache.struts.action.ActionServlet.process(ActionServlet.java:1196) at org.apache.struts.action.ActionServlet.doGet(ActionServlet.java:414) at javax.servlet.http.HttpServlet.service(HttpServlet.java:743) ... Caused by: java.lang.NumberFormatException: null at java.lang.Long.parseLong(Long.java:372) at java.lang.Long.parseLong(Long.java:461) at com.redhat.rhn.frontend.action.channel.manage.AddRedHatErrataAction.execute(AddRedHatErrataAction.java:83) at org.apache.struts.action.RequestProcessor.processActionPerform(RequestProcessor.java:431) ... 51 more Though looks like oracle/java's still working on it - top - 16:20:03 up 2:13, 2 users, load average: 3.14, 3.25, 3.46 Tasks: 97 total, 4 running, 92 sleeping, 0 stopped, 1 zombie Cpu(s): 77.7% us, 4.0% sy, 0.4% ni, 17.4% id, 0.5% wa, 0.0% hi, 0.0% si Mem: 2000004k total, 1736308k used, 263696k free, 29712k buffers Swap: 557048k total, 0k used, 557048k free, 1067228k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 2339 tomcat 16 0 527m 356m 13m S 99.7 18.3 82:36.41 java 2906 oracle 25 0 248m 86m 84m R 99.7 4.4 107:13.38 oracle rhn-java-sat-5.1.1/code/src/com/redhat/rhn/frontend/action/channel/manage/AddRedHatErrataAction.java ... private static final String CID = "cid"; ... public ActionForward execute(ActionMapping mapping, ActionForm formIn, HttpServletRequest request, HttpServletResponse response) { RequestContext requestContext = new RequestContext(request); User user = requestContext.getLoggedInUser(); Long cid = Long.parseLong(request.getParameter(CID)); <----- line 83 Channel currentChan = ChannelFactory.lookupByIdAndUser(cid, user); Channel selectedChannel = null; ... It looks like CID "cid" in the request is null (or at least it thinks it is)... Workarounds: 1. Cloning channels/errata (instead of adding) works around this problem; 2. Though I was never able to replicate the buffer overflow exception, it's possible it's related to http://www.nabble.com/is-there-a-hard-coded-size-limit-to-mod_jk-response-headers--td8350507.html which suggested configuring the packetSize at standard coyote AJP Jk handler on Tomcat 5.5.21 and above - http://svn.apache.org/viewvc/tomcat/container/tc5.5.x/webapps/docs/changelog.xml?r1=483339&r2=485795&diff_format=h Now, on internal 5.1 satellite I see tomcat5-5.0.30-0jpp_10rh and 5.2 satellite (el5) there's tomcat5-5.5.23-0jpp.7.el5_2.1 so looks like this is a configuration we can try out on satellite 5.2 or 5.3 beta as a possible tuning/workaround conf change. For Engineering: More details and reproducer info included in internal updates, if you need anything else, let me know. Thanks.
So flipping this to wont_fix as per discussion. If there is any issue with this, please re-open. Thanks
Reopening, more than one customer hitting it, so going to investigate further.
so this shouldn't be reproducible on 5.3, but an inefficient query will make this difficult to try to reproduce using steps in comment #1. This was fixed in spacewalk master in: 914db78b3a4f2a805e16914f62660bb7438ec11b
So this bug is actually present in 5.3.0 GA (and thus is not caused by this fix). There are two ways to get past this bug: a) use a cloned channel (not a plain custom channel) b) Click the "View Associated Errata" button before selecting the errata and hitting confirm This is fixed in spacewalk. Not sure if we want to fix this for this errata. I'll let cliff chime in...
Jiri, This is quite odd. It looked like perl-DateTime was not installed and that's what was causing the issue (I think). I installed that package on that box and the perl pages seem to be working. -Justin
rhel4 platform also ok, no error during cloning errata
An advisory has been issued which should help the problem described in this bug report. This report is therefore being closed with a resolution of ERRATA. For more information on therefore solution and/or where to find the updated files, please follow the link below. You may reopen this bug report if the solution does not work for you. http://rhn.redhat.com/errata/RHBA-2010-0021.html