Bug 1252142
Summary: | Creation and group association of bundle fails with timeout followed by IllegalArgumentException: Token did not result in valid file | ||||||||||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Product: | [JBoss] JBoss Operations Network | Reporter: | dsteigne | ||||||||||||||||||||
Component: | Provisioning | Assignee: | Jay Shaughnessy <jshaughn> | ||||||||||||||||||||
Status: | CLOSED ERRATA | QA Contact: | Filip Brychta <fbrychta> | ||||||||||||||||||||
Severity: | high | Docs Contact: | |||||||||||||||||||||
Priority: | urgent | ||||||||||||||||||||||
Version: | JON 3.3.3 | CC: | fbrychta, jshaughn, loleary, spinder | ||||||||||||||||||||
Target Milestone: | ER02 | Keywords: | Triaged | ||||||||||||||||||||
Target Release: | JON 3.3.4 | ||||||||||||||||||||||
Hardware: | Unspecified | ||||||||||||||||||||||
OS: | Unspecified | ||||||||||||||||||||||
Whiteboard: | |||||||||||||||||||||||
Fixed In Version: | Doc Type: | Bug Fix | |||||||||||||||||||||
Doc Text: | Story Points: | --- | |||||||||||||||||||||
Clone Of: | Environment: | ||||||||||||||||||||||
Last Closed: | 2015-10-28 14:37:15 UTC | Type: | Bug | ||||||||||||||||||||
Regression: | --- | Mount Type: | --- | ||||||||||||||||||||
Documentation: | --- | CRM: | |||||||||||||||||||||
Verified Versions: | Category: | --- | |||||||||||||||||||||
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||||||||||||||||
Cloudforms Team: | --- | Target Upstream Version: | |||||||||||||||||||||
Embargoed: | |||||||||||||||||||||||
Attachments: |
|
Description
dsteigne
2015-08-10 19:49:00 UTC
Created attachment 1061208 [details]
Printscreen of bundle creation error
Created attachment 1061209 [details]
Next button
My bundles are too large to attach to the bug, Build a bundle archive that is at least 300 MB in size. ~~~ tmpDir="`mktemp -d`" cat >"${tmpDir}"/deploy.xml <<EOF <?xml version="1.0"?> <project name="largeFile" default="main" xmlns:rhq="antlib:org.rhq.bundle"> <target name="main" /> <rhq:bundle name="Large File Test Bundle" version="1.0" description="A sample bundle consisting of a single large file"> <rhq:deployment-unit name="files" compliance="filesAndDirectories"> EOF mkdir "${tmpDir}"/files for file in {001..001}; do echo "This is File #${file}." > "${tmpDir}/files/file-${file}.txt" dd if=/dev/urandom of="${tmpDir}/files/file-${file}.bin" bs=30M count=10 echo ' <rhq:file name="files/file-'"${file}"'.txt" destinationFile="files/file-'"${file}"'.txt" replace="true" />' >> "${tmpDir}"/deploy.xml echo ' <rhq:file name="files/file-'"${file}"'.bin" destinationFile="files/file-'"${file}"'.bin" replace="true" />' >> "${tmpDir}"/deploy.xml done cat >>"${tmpDir}"/deploy.xml <<EOF </rhq:deployment-unit> </rhq:bundle> </project> EOF (cd "${tmpDir}" && zip -r /tmp/large-bundle.zip .) rm -rf "${tmpDir}" ls -al /tmp/large-bundle.zip ~~~ This can't be about user rights, I could successfully upload with those rights the bundle. I can't actually verify this bug at all, I can upload with the same rights a 1GB bundle successfully. There's one thing though that might be the reason, the bundle binary is stored in the database. Maybe the database can't handle this load or is so slow the query timeout occurs.. mm. I don't have enough slow machine to verify this. (In reply to Michael Burman from comment #7) > This can't be about user rights, I could successfully upload with those > rights the bundle. I can't actually verify this bug at all, I can upload > with the same rights a 1GB bundle successfully. > > There's one thing though that might be the reason, the bundle binary is > stored in the database. Maybe the database can't handle this load or is so > slow the query timeout occurs.. mm. I don't have enough slow machine to > verify this. Michael, Please let me know what permissions you have set on your test user. Created attachment 1078776 [details]
server log from my test
Created attachment 1078778 [details]
Bundle user's permissions
I am able to reproduce on my version 3.3.3 system with a 291MB bundle, I've attached my server log from the time and the permissions for the user uploading the bundle. Again if the user has full permissions there is no issue. Created attachment 1079057 [details]
Latest Server.log from test
commit 1990e7d656006d8e868a1de7b20d12ec4b2d6ea0 Author: Jay Shaughnessy <jshaughn> Date: Fri Oct 2 15:40:47 2015 -0400 BZ-1252142 Creation and group association of bundle fails with timeout See BZ for a detailed description of the issue. This commit basically increases the timeout to compensate for the longer time required to stream large bundles into the DB, when performed via the GUI. Moving to ON_QA as available to test with the following build: https://brewweb.devel.redhat.com/buildinfo?buildID=460382 *Note: jon-server-patch-3.3.0.GA.zip maps to ER01 build of jon-server-3.3.0.GA-update-04.zip. Original problem is resolved but the wizard is still not completed successfully. The bundle created as described in comment 3 is successfully uploaded, but assignment to the group step fails because of timeout. See attached screen shots - fist click on Next button, second click on Next button and third click on Next button. Server log is attached as well. Created attachment 1081308 [details]
screenshot after first click
Created attachment 1081309 [details]
screenshot after second click
Created attachment 1081310 [details]
screenshot after third click
Created attachment 1081311 [details]
server log
This is still showing a 30s timeout. I don't understand why that would be. Looking into it... MASTER commit 86bbcd55209661e48beefc987dc3532142267930 Author: Jay Shaughnessy <jshaughn> Date: Fri Oct 9 16:29:13 2015 -0400 The original fix commit for this problem is valid (1990e7d65). It increased the RPC timeout from the default, 30s, to 10m in order to provide ample time for large distribution files to be processed server-side. This uncovered an unrelated issue with the way our RPC timeouts are handled. In short, gwt keeps only one instance of each rpc service even though you can reassign [custom] request builders, which we do, to apply our timeouts to the service. But each service lookup reassigns the timeout to it's needs. So if you lookup the service and assign a 10m timeout, it can be 'overwritten' before it's actually used. And this is what happened in the QA test. I believe it could behave differently on different systems. On my system I actually saw it overwritten twice between two consecutive lines of code, the first looking up the bundle service and the second making the call. The reason is that we don't control the order of everything going on in gwt. For example, several classes initialized the bundle service as a class-field. The initialization could happen at unexpected times. Instead, by chaining every bundle-service-lookup to the service call, the problem has gone away. I particularly suspect the class-level decls in the DataSources, but we had plenty of advance initializations going on. This problem could may still affect other code using other services. This commit is for the bundle service only, other changes were out of scope. branch: release/jon3.3.x commit: aa79ebdd9d7b2502123b92ce9945a1a079f4528f Author: Jay Shaughnessy <jshaughn> AuthorDate: Fri Oct 9 16:29:13 2015 -0400 Commit: Simeon Pinder <spinder> CommitDate: Tue Oct 13 07:08:13 2015 -0400 [BZ 1252142] Creation and group association of bundle fails with timeout followed by IllegalArgumentException Moving to ON_QA as available to test with the following build: https://brewweb.devel.redhat.com//buildinfo?buildID=461043 *Note: jon-server-patch-3.3.0.GA.zip maps to ER02 build of jon-server-3.3.0.GA-update-04.zip. Verified on: Version : 3.3.0.GA Update 04 Build Number : e9ed05b:aa79ebd Found two related issues: bz1272473 bz1272358 Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHSA-2015-1947.html |