Bug 848344 - Problem submitting jobs from cumin via Aviary when commands have no arguments
Problem submitting jobs from cumin via Aviary when commands have no arguments
Status: CLOSED ERRATA
Product: Red Hat Enterprise MRG
Classification: Red Hat
Component: cumin (Show other bugs)
2.1.2
Unspecified Unspecified
low Severity medium
: 2.3
: ---
Assigned To: Trevor McKay
Daniel Horák
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2012-08-15 06:00 EDT by Daniel Horák
Modified: 2013-03-06 13:45 EST (History)
4 users (show)

See Also:
Fixed In Version: cumin-0.1.5668-1
Doc Type: Bug Fix
Doc Text:
Cause The Aviary web service requires command arguments for all jobs submitted in cumin via Aviary. Cumin does not supply a command arguments value to Aviary if the user does not specify arguments. This is a problem for commands that take no arguments. Consequence Submissions that specify commands without arguments will be rejected with an error. This applies to regular submissions and VM submissions. Fix: Cumin supplies a null command arguments value to Aviary when no command arguments are specified by the user. Result It is possible to successfully submit commands with no arguments as jobs from cumin.
Story Points: ---
Clone Of:
Environment:
Last Closed: 2013-03-06 13:45:06 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
Related part from web.log (3.59 KB, text/x-log)
2012-08-15 06:00 EDT, Daniel Horák
no flags Details
Patch to handle missing fields in Aviary job submission (753 bytes, patch)
2012-08-15 11:04 EDT, Trevor McKay
no flags Details | Diff

  None (edit)
Description Daniel Horák 2012-08-15 06:00:18 EDT
Created attachment 604571 [details]
Related part from web.log

Description of problem:
  I'm not able to submit VM job from cumin via aviary. Submitting ends with:
    Submit VM job 'VM cumin job': Failed ('Args')
  And in web.log is following error:
    29483 2012-08-15 11:38:25,803 ERROR 'Args'
    Traceback (most recent call last):
      File "/usr/share/cumin/python/cumin/task.py", line 98, in invoke
        self.do_invoke(session, obj, invoc, *args, **kwargs)
      File "/usr/share/cumin/python/cumin/grid/submission.py", line 720, in do_invoke
        
      File "/usr/share/cumin/python/sage/aviary/aviaryoperations.py", line 152, in submit_job
        args.append(ad[attr])
    KeyError: 'Args'


Version-Release number of selected component (if applicable):

# rpm -qa | grep -e cumin -e condor
  condor-7.6.5-0.20.el5.i386
  condor-aviary-7.6.5-0.20.el5.i386
  condor-classads-7.6.5-0.20.el5.i386
  condor-qmf-7.6.5-0.20.el5.i386
  cumin-0.1.5444-2.el5.noarch



How reproducible:
  100%


Steps to Reproduce:
1. Install and configure cumin to connect to condor pool with VM universe configured (via aviary).
2. Try to submit VM job via cumin (I filled only description and vm image location).
  
Actual results:
  Submission failed with following error and error message in web.log:
    Submit VM job 'VM cumin job': Failed ('Args')
  

Expected results:
  Job is correctly submitted and no error appear.


Additional info:
Comment 1 Trevor McKay 2012-08-15 09:50:14 EDT
This is actually a general problem, not specific to VM jobs.

The same failure can be seen when submitting a normal job with no command arguments ("/bin/sleep" in the command field rather than "/bin/sleep 1d")
Comment 3 Trevor McKay 2012-08-15 11:04:29 EDT
Created attachment 604625 [details]
Patch to handle missing fields in Aviary job submission
Comment 8 Trevor McKay 2012-08-16 13:02:51 EDT
Setting needinfo on dahorak.

Daniel, I think I've found a workaround.  Please confirm.

  If you expand the form for a regular submission or a VM submission and add a blank Args field in extra attributes, it will work.  

Extra attributes should look like this, with nothing after the =:

Args =

 This should get the job out of Cumin, and it will not be passed any arguments when it runs.
Comment 10 Daniel Horák 2012-08-16 15:05:36 EDT
(In reply to comment #8)
> If you expand the form for a regular submission or a VM submission and add
> a blank Args field in extra attributes, it will work.  
> 
> Extra attributes should look like this, with nothing after the =:
> 
> Args =
> 

I checked this workaround and it is working.
Comment 12 Trevor McKay 2012-08-16 15:31:52 EDT
    Technical note added. If any revisions are required, please edit the "Technical Notes" field
    accordingly. All revisions will be proofread by the Engineering Content Services team.
    
    New Contents:
Cause
    The CuminAviary Tech Preview feature requires command arguments for jobs submitted in Cumin via Aviary.

Consequence
    Submissions that specify commands without arguments will be rejected by Cumin with an error.  This applies to regular submissions and VM submissions.

Workaround
    Expand the submission form via the "Show more" button and add the string "Args =" to the "Extra attributes" field. This will satisfy the restriction in Cumin without defining any actual arguments.  The job will run in condor without any arguments being passed to the command.

Result
    This workaround should be 100% effective for submitting jobs without command arguments via Aviary.
Comment 13 Trevor McKay 2012-08-16 15:34:57 EDT
This is fixed in revision 5445 on trunk
Comment 14 Trevor McKay 2012-10-03 16:08:08 EDT
Testing

This can be done with a simple (non-vm) job since this is a general problem in the aviaryoperations code

1) Choose a command that legitimately takes no arguments.  I used /bin/date.

2) Create a simple submission with "Submit job"

3) This command will run very quickly, and may not even be visible in condor_q to verify that it's running. So, click the "Show more" button and set "Standard output" to "/tmp/mytest".  You may also set "Standard error" and "User Log" to files in /tmp.

4) Submit the job.  It should go "Pending" and then "OK"

5) Check /tmp/mytest for output.  In the case of /bin/date, there should be a date string in the file to confirm that the command ran.
Comment 16 Daniel Horák 2013-01-28 07:29:20 EST
I seems like there is some new problem - the logs and messages are different, but it still don't work.

Submitting job without arguments leads to error message in cumin:
  2013-01-28 13:21:09  Submit job 'Job without arguments 2': Failed (Operation failed on host HOSTNAME, (u"Server raised fault: 'Data element of the OM Node is NULL'",))

Related part of web.log:
# tail -F /var/log/cumin/web.log
  4559 2013-01-28 13:21:09,257 INFO Request POST /form.html?
  4559 2013-01-28 13:21:09,258 DEBUG Validating cumin.grid.submission.JobSubmitForm('modes.JobSubmit')
  4559 2013-01-28 13:21:09,261 DEBUG Starting cumin.grid.submission.JobSubmit
  4559 2013-01-28 13:21:09,261 INFO Started cumin.grid.submission.JobSubmit
  4559 2013-01-28 13:21:09,261 DEBUG Job ad:
  4559 2013-01-28 13:21:09,261 DEBUG   !!descriptors                       {'Requirements': 'com.redhat.grid.Expression'}
  4559 2013-01-28 13:21:09,262 DEBUG   Cmd                                 '/bin/date'
  4559 2013-01-28 13:21:09,262 DEBUG   DiskUsage                           0
  4559 2013-01-28 13:21:09,262 DEBUG   Err                                 '/tmp/mytest.err'
  4559 2013-01-28 13:21:09,262 DEBUG   Iwd                                 '/tmp'
  4559 2013-01-28 13:21:09,262 DEBUG   Out                                 '/tmp/mytest.out'
  4559 2013-01-28 13:21:09,263 DEBUG   Owner                               'cumin'
  4559 2013-01-28 13:21:09,263 DEBUG   RequestDisk                         1048576
  4559 2013-01-28 13:21:09,263 DEBUG   RequestMemory                       512
  4559 2013-01-28 13:21:09,263 DEBUG   Requirements                        '(FileSystemDomain =!= UNDEFINED && Arch =!= UNDEFINED)'
  4559 2013-01-28 13:21:09,263 DEBUG   Submission                          'Job without arguments 2'
  4559 2013-01-28 13:21:09,264 DEBUG   UserLog                             '/tmp/mytest.log'
  4559 2013-01-28 13:21:09,270 DEBUG Exiting cumin.grid.submission.JobSubmit
  4559 2013-01-28 13:21:09,270 INFO Exited cumin.grid.submission.JobSubmit
  4559 2013-01-28 13:21:09,271 INFO Response 303 See Other
  4559 2013-01-28 13:21:09,271 DEBUG Response headers:
  4559 2013-01-28 13:21:09,271 DEBUG   Location                  index.html?frame=main.grid;main.grid.id=1;main.grid.view.body.m=pool_submissions
  4559 2013-01-28 13:21:09,276 INFO Request GET /index.html?frame=main.grid;main.grid.id=1;main.grid.view.body.m=pool_submissions
  4559 2013-01-28 13:21:09,294 INFO Response 200 OK
  4559 2013-01-28 13:21:09,294 DEBUG Response headers:
  4559 2013-01-28 13:21:09,294 DEBUG   Content-Length            16811
  4559 2013-01-28 13:21:09,294 DEBUG   Content-Type              application/xhtml+xml
  4559 2013-01-28 13:21:09,308 DEBUG Ending cumin.grid.submission.JobSubmit
  4559 2013-01-28 13:21:09,308 INFO Ended cumin.grid.submission.JobSubmit

Behaviour for VM universe job is similar.
Workaround for MRG 2.2. from comment 8 works well.

>> back to ASSIGNED
Comment 17 Trevor McKay 2013-01-28 10:52:50 EST
My fix for this issue is incorrect.

Instead of an exception occurring in cumin, badly constructed messsages are being passed to Aviary and the server is raising an exception.  So the error is different but overall result is the same.

Summary of current status: this is a known issue since 2.1, it is still broken, and there is a known workaround.
Comment 19 Daniel Horák 2013-02-04 08:48:39 EST
Retested and verified on RHEL 5.9/6.4 - i386/x86_64.

# rpm -qa | grep -e condor -e cumin -e qpid | sort
  condor-7.8.8-0.4.1.el5.i386
  condor-aviary-7.8.8-0.4.1.el5.i386
  condor-classads-7.8.8-0.4.1.el5.i386
  condor-qmf-7.8.8-0.4.1.el5.i386
  cumin-0.1.5668-1.el5.noarch
  python-qpid-0.18-4.el5.noarch
  python-qpid-qmf-0.18-14.el5.i386
  qpid-cpp-client-0.18-14.el5.i386
  qpid-cpp-server-0.18-14.el5.i386
  qpid-qmf-0.18-14.el5.i386
  qpid-tools-0.18-7.el5.noarch

>>> VERIFIED
Comment 20 Trevor McKay 2013-02-15 11:54:51 EST
Failed to mention that this issue was fixed in revision 5668.
Comment 22 errata-xmlrpc 2013-03-06 13:45:06 EST
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHSA-2013-0564.html

Note You need to log in before you can comment on or make changes to this bug.