Bug 848344
| Summary: | Problem submitting jobs from cumin via Aviary when commands have no arguments | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| Product: | Red Hat Enterprise MRG | Reporter: | Daniel Horák <dahorak> | ||||||
| Component: | cumin | Assignee: | Trevor McKay <tmckay> | ||||||
| Status: | CLOSED ERRATA | QA Contact: | Daniel Horák <dahorak> | ||||||
| Severity: | medium | Docs Contact: | |||||||
| Priority: | low | ||||||||
| Version: | 2.1.2 | CC: | ltoscano, matt, sgraf, tmckay | ||||||
| Target Milestone: | 2.3 | ||||||||
| Target Release: | --- | ||||||||
| Hardware: | Unspecified | ||||||||
| OS: | Unspecified | ||||||||
| Whiteboard: | |||||||||
| Fixed In Version: | cumin-0.1.5668-1 | Doc Type: | Bug Fix | ||||||
| Doc Text: |
Cause
The Aviary web service requires command arguments for all jobs submitted in cumin via Aviary. Cumin does not supply a command arguments value to Aviary if the user does not specify arguments. This is a problem for commands that take no arguments.
Consequence
Submissions that specify commands without arguments will be rejected with an error. This applies to regular submissions and VM submissions.
Fix:
Cumin supplies a null command arguments value to Aviary when no command arguments are specified by the user.
Result
It is possible to successfully submit commands with no arguments as jobs from cumin.
|
Story Points: | --- | ||||||
| Clone Of: | Environment: | ||||||||
| Last Closed: | 2013-03-06 18:45:06 UTC | Type: | Bug | ||||||
| Regression: | --- | Mount Type: | --- | ||||||
| Documentation: | --- | CRM: | |||||||
| Verified Versions: | Category: | --- | |||||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||||
| Embargoed: | |||||||||
| Attachments: |
|
||||||||
This is actually a general problem, not specific to VM jobs.
The same failure can be seen when submitting a normal job with no command arguments ("/bin/sleep" in the command field rather than "/bin/sleep 1d")
Created attachment 604625 [details]
Patch to handle missing fields in Aviary job submission
Setting needinfo on dahorak. Daniel, I think I've found a workaround. Please confirm. If you expand the form for a regular submission or a VM submission and add a blank Args field in extra attributes, it will work. Extra attributes should look like this, with nothing after the =: Args = This should get the job out of Cumin, and it will not be passed any arguments when it runs. (In reply to comment #8) > If you expand the form for a regular submission or a VM submission and add > a blank Args field in extra attributes, it will work. > > Extra attributes should look like this, with nothing after the =: > > Args = > I checked this workaround and it is working.
Technical note added. If any revisions are required, please edit the "Technical Notes" field
accordingly. All revisions will be proofread by the Engineering Content Services team.
New Contents:
Cause
The CuminAviary Tech Preview feature requires command arguments for jobs submitted in Cumin via Aviary.
Consequence
Submissions that specify commands without arguments will be rejected by Cumin with an error. This applies to regular submissions and VM submissions.
Workaround
Expand the submission form via the "Show more" button and add the string "Args =" to the "Extra attributes" field. This will satisfy the restriction in Cumin without defining any actual arguments. The job will run in condor without any arguments being passed to the command.
Result
This workaround should be 100% effective for submitting jobs without command arguments via Aviary.
This is fixed in revision 5445 on trunk Testing This can be done with a simple (non-vm) job since this is a general problem in the aviaryoperations code 1) Choose a command that legitimately takes no arguments. I used /bin/date. 2) Create a simple submission with "Submit job" 3) This command will run very quickly, and may not even be visible in condor_q to verify that it's running. So, click the "Show more" button and set "Standard output" to "/tmp/mytest". You may also set "Standard error" and "User Log" to files in /tmp. 4) Submit the job. It should go "Pending" and then "OK" 5) Check /tmp/mytest for output. In the case of /bin/date, there should be a date string in the file to confirm that the command ran. I seems like there is some new problem - the logs and messages are different, but it still don't work.
Submitting job without arguments leads to error message in cumin:
2013-01-28 13:21:09 Submit job 'Job without arguments 2': Failed (Operation failed on host HOSTNAME, (u"Server raised fault: 'Data element of the OM Node is NULL'",))
Related part of web.log:
# tail -F /var/log/cumin/web.log
4559 2013-01-28 13:21:09,257 INFO Request POST /form.html?
4559 2013-01-28 13:21:09,258 DEBUG Validating cumin.grid.submission.JobSubmitForm('modes.JobSubmit')
4559 2013-01-28 13:21:09,261 DEBUG Starting cumin.grid.submission.JobSubmit
4559 2013-01-28 13:21:09,261 INFO Started cumin.grid.submission.JobSubmit
4559 2013-01-28 13:21:09,261 DEBUG Job ad:
4559 2013-01-28 13:21:09,261 DEBUG !!descriptors {'Requirements': 'com.redhat.grid.Expression'}
4559 2013-01-28 13:21:09,262 DEBUG Cmd '/bin/date'
4559 2013-01-28 13:21:09,262 DEBUG DiskUsage 0
4559 2013-01-28 13:21:09,262 DEBUG Err '/tmp/mytest.err'
4559 2013-01-28 13:21:09,262 DEBUG Iwd '/tmp'
4559 2013-01-28 13:21:09,262 DEBUG Out '/tmp/mytest.out'
4559 2013-01-28 13:21:09,263 DEBUG Owner 'cumin'
4559 2013-01-28 13:21:09,263 DEBUG RequestDisk 1048576
4559 2013-01-28 13:21:09,263 DEBUG RequestMemory 512
4559 2013-01-28 13:21:09,263 DEBUG Requirements '(FileSystemDomain =!= UNDEFINED && Arch =!= UNDEFINED)'
4559 2013-01-28 13:21:09,263 DEBUG Submission 'Job without arguments 2'
4559 2013-01-28 13:21:09,264 DEBUG UserLog '/tmp/mytest.log'
4559 2013-01-28 13:21:09,270 DEBUG Exiting cumin.grid.submission.JobSubmit
4559 2013-01-28 13:21:09,270 INFO Exited cumin.grid.submission.JobSubmit
4559 2013-01-28 13:21:09,271 INFO Response 303 See Other
4559 2013-01-28 13:21:09,271 DEBUG Response headers:
4559 2013-01-28 13:21:09,271 DEBUG Location index.html?frame=main.grid;main.grid.id=1;main.grid.view.body.m=pool_submissions
4559 2013-01-28 13:21:09,276 INFO Request GET /index.html?frame=main.grid;main.grid.id=1;main.grid.view.body.m=pool_submissions
4559 2013-01-28 13:21:09,294 INFO Response 200 OK
4559 2013-01-28 13:21:09,294 DEBUG Response headers:
4559 2013-01-28 13:21:09,294 DEBUG Content-Length 16811
4559 2013-01-28 13:21:09,294 DEBUG Content-Type application/xhtml+xml
4559 2013-01-28 13:21:09,308 DEBUG Ending cumin.grid.submission.JobSubmit
4559 2013-01-28 13:21:09,308 INFO Ended cumin.grid.submission.JobSubmit
Behaviour for VM universe job is similar.
Workaround for MRG 2.2. from comment 8 works well.
>> back to ASSIGNED
My fix for this issue is incorrect. Instead of an exception occurring in cumin, badly constructed messsages are being passed to Aviary and the server is raising an exception. So the error is different but overall result is the same. Summary of current status: this is a known issue since 2.1, it is still broken, and there is a known workaround. Retested and verified on RHEL 5.9/6.4 - i386/x86_64.
# rpm -qa | grep -e condor -e cumin -e qpid | sort
condor-7.8.8-0.4.1.el5.i386
condor-aviary-7.8.8-0.4.1.el5.i386
condor-classads-7.8.8-0.4.1.el5.i386
condor-qmf-7.8.8-0.4.1.el5.i386
cumin-0.1.5668-1.el5.noarch
python-qpid-0.18-4.el5.noarch
python-qpid-qmf-0.18-14.el5.i386
qpid-cpp-client-0.18-14.el5.i386
qpid-cpp-server-0.18-14.el5.i386
qpid-qmf-0.18-14.el5.i386
qpid-tools-0.18-7.el5.noarch
>>> VERIFIED
Failed to mention that this issue was fixed in revision 5668. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. http://rhn.redhat.com/errata/RHSA-2013-0564.html |
Created attachment 604571 [details] Related part from web.log Description of problem: I'm not able to submit VM job from cumin via aviary. Submitting ends with: Submit VM job 'VM cumin job': Failed ('Args') And in web.log is following error: 29483 2012-08-15 11:38:25,803 ERROR 'Args' Traceback (most recent call last): File "/usr/share/cumin/python/cumin/task.py", line 98, in invoke self.do_invoke(session, obj, invoc, *args, **kwargs) File "/usr/share/cumin/python/cumin/grid/submission.py", line 720, in do_invoke File "/usr/share/cumin/python/sage/aviary/aviaryoperations.py", line 152, in submit_job args.append(ad[attr]) KeyError: 'Args' Version-Release number of selected component (if applicable): # rpm -qa | grep -e cumin -e condor condor-7.6.5-0.20.el5.i386 condor-aviary-7.6.5-0.20.el5.i386 condor-classads-7.6.5-0.20.el5.i386 condor-qmf-7.6.5-0.20.el5.i386 cumin-0.1.5444-2.el5.noarch How reproducible: 100% Steps to Reproduce: 1. Install and configure cumin to connect to condor pool with VM universe configured (via aviary). 2. Try to submit VM job via cumin (I filled only description and vm image location). Actual results: Submission failed with following error and error message in web.log: Submit VM job 'VM cumin job': Failed ('Args') Expected results: Job is correctly submitted and no error appear. Additional info: