Description of problem: Submit a vmcore with an explicit kernel version results in Internal Server Error The server encountered an internal error or misconfiguration and was unable to complete your request. Please contact the server administrator, root@localhost and inform them of the time the error occurred, and anything you might have done that may have caused the error. More information about this error may be available in the server error log. Version-Release number of selected component (if applicable): retrace-server-1.12-2.el6.noarch How reproducible: As far as I can tell it happens every time with any vmcore and any kernel version when using the 'manager' web UI or 'Custom Core Location' to submit a vmcore. Steps to Reproduce: 1. Submit a vmcore via the WEB UI using a specific kernel version such as 2.6.32-220.el6.x86_64 Actual results: Internal Server Error The server encountered an internal error or misconfiguration and was unable to complete your request. Please contact the server administrator, root@localhost and inform them of the time the error occurred, and anything you might have done that may have caused the error. More information about this error may be available in the server error log. Expected results: No error Additional info: Unfortunately due to this problem, we had to roll back to retrace-server-1.11-4.el6.noarch on our production retrace-server instance. The only thing I could find was in the apache log: # tail /var/log/httpd/error_log [Mon Sep 22 14:37:49 2014] [error] [client 10.10.52.30] mod_wsgi (pid=20566): Exception occurred processing WSGI script '/usr/share/retrace-server/manager.wsgi'. [Mon Sep 22 14:37:49 2014] [error] [client 10.10.52.30] Traceback (most recent call last): [Mon Sep 22 14:37:49 2014] [error] [client 10.10.52.30] File "/usr/share/retrace-server/manager.wsgi", line 125, in application [Mon Sep 22 14:37:49 2014] [error] [client 10.10.52.30] arch = kernelver.arch [Mon Sep 22 14:37:49 2014] [error] [client 10.10.52.30] AttributeError: 'str' object has no attribute 'arch' [Mon Sep 22 14:38:43 2014] [error] [client 10.10.52.30] mod_wsgi (pid=20566): Exception occurred processing WSGI script '/usr/share/retrace-server/manager.wsgi'. [Mon Sep 22 14:38:43 2014] [error] [client 10.10.52.30] Traceback (most recent call last): [Mon Sep 22 14:38:43 2014] [error] [client 10.10.52.30] File "/usr/share/retrace-server/manager.wsgi", line 125, in application [Mon Sep 22 14:38:43 2014] [error] [client 10.10.52.30] arch = kernelver.arch [Mon Sep 22 14:38:43 2014] [error] [client 10.10.52.30] AttributeError: 'str' object has no attribute 'arch'
The most obvious bug to check would be this patch to clarify 'kernel VRA': https://bugzilla.redhat.com/show_bug.cgi?id=1080702#c2
(In reply to Dave Wysochanski from comment #1) > The most obvious bug to check would be this patch to clarify 'kernel VRA': > https://bugzilla.redhat.com/show_bug.cgi?id=1080702#c2 The above doesn't look relevant. FWIW, from what I can tell, something in this list caused the regression: a12f064 increase version to 1.12 dc055c6 rs-interact: set finish time if necessary f4cffef rs-cleanup: add config sanity checks e9dbe45 add kernel version example where applicable 4cdd713 forward kwargs in task.start() d8168b6 rs-interact: add 'set-success' and 'set-fail' actions 91a6d6c add arch-based remote execution 42db064 retrace-server.conf: clarify cleanup options 134f279 get_kernel_release: do not match 'OSRELEASE=%s' Just guessing but probably ones to look at: commit 4cdd713d73dde059593d37fca553fb3bc9fb7492 Author: Michal Toman <mtoman> Date: Wed Jul 30 10:45:59 2014 +0200 forward kwargs in task.start() Signed-off-by: Michal Toman <mtoman> commit 91a6d6cd772cfb950f933a032a3212f80cf2e514 Author: Michal Toman <mtoman> Date: Wed Jun 11 14:34:15 2014 +0200 add arch-based remote execution Signed-off-by: Michal Toman <mtoman>
https://docs.python.org/2/library/functions.html#str "Return a string containing a nicely printable representation of an object." ------------------------------------------------------- commit 91a6d6cd772cfb950f933a032a3212f80cf2e514 Author: Michal Toman <mtoman> Date: Wed Jun 11 14:34:15 2014 +0200 add arch-based remote execution Signed-off-by: Michal Toman <mtoman> @@ -122,15 +121,13 @@ def application(environ, start_response): except Exception as ex: return response(start_response, "403 Forbidden", _("Please use - cmdline.append("--kernelver") - cmdline.append(str(kernelver)) - cmdline.append("--arch") - cmdline.append(kernelver.arch) + kernelver = str(kernelver) + arch = kernelver.arch if "notify" in get: task.set_notify(filter(None, set(n.strip() for n in get["notify"][0 - call(cmdline) + task.start(debug=debug, kernelver=kernelver, arch=arch) # ugly, ugly, ugly! retrace-server-worker double-forks and needs a whil time.sleep(2) ------------------------------------------------------- Fix?: ------------------------------------------------------- From d93dd3e8a7b00823541b7cec4431a11841ea363c Mon Sep 17 00:00:00 2001 From: Harshula Jayasuriya <harshula> Date: Tue, 23 Sep 2014 11:49:00 +1000 Subject: [PATCH] manager.wsgi: Don't treat a string as an object Error: Traceback (most recent call last): File "/usr/share/retrace-server/manager.wsgi", line 125, in application arch = kernelver.arch AttributeError: 'str' object has no attribute 'arch' Grab the arch from kernelver before it's made into a string. --- src/manager.wsgi | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/manager.wsgi b/src/manager.wsgi index fcd4b7e..d1765f3 100644 --- a/src/manager.wsgi +++ b/src/manager.wsgi @@ -121,8 +121,8 @@ def application(environ, start_response): except Exception as ex: return response(start_response, "403 Forbidden", _("Please use VRA format for kernel version (e.g. 2.6.32-287.el6.x86_64)")) - kernelver = str(kernelver) arch = kernelver.arch + kernelver = str(kernelver) if "notify" in get: task.set_notify(filter(None, set(n.strip() for n in get["notify"][0].replace(";", ",").split(",")))) -- 1.9.3 -------------------------------------------------------
Created attachment 940403 [details] Don't treat a string as an object
(In reply to Harshula Jayasuriya from comment #4) > Created attachment 940403 [details] > Don't treat a string as an object Thx Harshula! Need more sleep, and/or read the apache error log more carefully! And FWIW, verified on optimus-dev just now. Acked-by: Dave Wysochanski <dwysocha>
Michal - can we get this 1-line fix into a retrace-server build (maybe 1.12-3.el6) so we can get the latest code into production? Or do you have other concerns about some of the patches?
Sorry for not looking at this earlier. The patch looks good, I've merged it upstream.
(In reply to Michal Toman from comment #7) > Sorry for not looking at this earlier. The patch looks good, I've merged it > upstream. No problem - thanks for merging. Will do sanity testing with retrace-server-1.12-3.el6.noarch.rpm in dev then move to production ASAP.
Looks fixed in retrace-server-1.12-3.el6.noarch.rpm
The fix is definitely committed in upstream: commit 702a09da52b59edf712c23e80770964af4381688 Author: Harshula Jayasuriya <harshula> Date: Tue Sep 23 11:49:00 2014 +1000 manager.wsgi: Don't treat a string as an object Error: Traceback (most recent call last): File "/usr/share/retrace-server/manager.wsgi", line 125, in application arch = kernelver.arch AttributeError: 'str' object has no attribute 'arch' Grab the arch from kernelver before it's made into a string. Dave's confirmed that it is fixed in retrace-server-1.12-3.el6.noarch.rpm. Is that everything, should this BZ be closed now?